KNIME: duplicate column issue

Sometimes if you try to read in multiple SDFs with the "SDF Reader" node and "extract all properties" at the same time (in order to extract all the associated tag data) the node can fail with:

"Execute failed: Duplicate column name "<colname>" at positions 1 and 2" - (the column names and reported positions depend on the format of the input data).

I tried to fix up the columns thinking this was a whitespace issue or some such but had no joy. For me the solution was to read the molecules without extracting the data and then use the "SDF Extractor" node to grab the data. This worked with no issues.

ShrewSoft VPN Manager window does not show under 64Bit Windows 7

I'm using version 2.2.2 of the ShrewSoft VPN client on my 64bit Win7 laptop. For some reason, the 'Manage' window doesn't show in this version - which bit me when my VPN details changed. The .pcf configuration files aren't re-read if you edit them and so you have to have access to the 'manage' dialog in order to update your connection - you can't even add a connection without accessing this dialog.

So, the solution for me was actually quite simple -

1. try to bring up the 'Manager' dialog (right click the ShreSoft icon in the bottom right of the taskbar and click on 'Manage')
2. bring up the task manager (Ctrl-Shift-Esc is a handy short-cut for that) - the VPN manager should be present in the "Applications" window
3. right click this and select 'maximize'

It might not be pretty, but at least it's now accessible.

Compare a local and a remote file using 'diff' and process substitution.

I don't think I've posted this one before: I've been doing a lot of moving files around between local and remote servers and have made heavy use of the above idiom. First find the files that need to be manually inspected (rsync -ilvrn <localdir> <remotedir> ) and then use the above diff command to compare them.

Better tab use with bash

I use tab delimited files a lot. I hate comma separated values (and if you've ever had to deal with fields that could contain commas, you'll agree). If you like to use a good set of linux command line tools, then tabs sometimes cause problems.

One of the most viewed posts on this blog is the unix-join-with-tabs one where I describe the "Ctrl-v " method of inserting a tab character on the command line for things like cut -f':' --output-delimiter=<tab> where the output delimiter needs to be a tab.

However, there's a much nicer/easier way of specifying a tab character in bash: $'\t'. Take a look a the QUOTING section of the bash manpage for all the details.

Here's a simple example of it in action:

Using 'find' to list files with multiple suffixes

I'm going through another data cleanup session in a (very) old work directory tree. I found myself examining/compressing/removing files with a recurring set of suffixes. The 'find' command can make this all a lot less painful.

Here's the command for matching a single suffix:

But editing this command-line to match the next suffix of interest becomes tedious very quickly. Thankfully, you can chain together file tests like so (note the grouping):

Then acting on these files is easy - just update the -exec action to what you want (e.g. "-exec bzip2 {} \;" - you probably want to use xargs or "-exec bzip2 {} +" for this to reduce the number of command invocations)

An interesting note here is that the following command isn't executed as you might expect. The 'ls' command is only executed on the *.tsv files due to the way the expression is evaluated: from left to right with the implicit '-and' between the second '-name' and '-exec' exec having higher precedence than the '-or' between the two '-name' tests..

Finding (and fixing) files with undesirable permissions

I use a few programs that create files that are not group read/writable; this is an issue when working in a group environment (especially with Knime where important files are locked this way which prevents other users from even opening your workspaces). Here's a find command that'll locate these files: This command executes 'ls -lh' on each of the files to show the current permissions (would probably be more efficient to push into xargs to run ls on multiple files at once). The next find command will update the permissions on those files: Of course, you can just run chmod recursively (chmod -R g+rw *), but this isn't always what you want.

Merge images

I have a couple of sets of related density plots (each set with ~9 images) that I need to enter into a report. I'm feeling lazy today and don't want to go through the pain of uploading each of these to our internal wiki so I merged them instead.

The montage program (part of the ImageMagick suite of tools) is really simple and effective for this:

Hey presto a set of merged density plots laid out into two columns (regardless of the number of images) with some spacing between them and a bit of drop shadow to visually separate the plots.