OpenBabel: Convert SDF to SMILES and keep the data!

This seems like it should be the default option when converting from an SDF to a SMILES file - keep the damned data! Well, in openbabel (at least the version I'm running) this is not the default. If you want to keep the SD data you have to specify each tag as part of the '--append' argument. e.g.

> babel test1.sdf --append "cLogD7.4 cLogP model_score1 model_score2 some_other_property" test1.smi

In order to end up with a tab delimited file (my favourite) then you have to prefix the argument to 'append' with the desired character. I used "Ctrl-v <tab>" to get a tab in my string. Seems odd that tabs wouldn't be the default delimiter since there's still a tab used to separate the SMILES string from the molecule name in the standard conversion.

simple parallel processing with make

I've used xargs a fair bit for some simple, local, parallel processing. I was just recently reminded of another clever and simple solution that a friend came up with.

I'll let the code do the talking; here's the basic bash script (stored as an executable):

#!/bin/bash

if [[ $# -ne 1 ]]; then
   echo "Usage: cat commands.txt | $(basename $0) <num processes>"
   exit 1
fi

(while read line; do
  echo -e "$((++i)):\n\t$line";
done; 
echo "all:" $(seq 1 $i)) | make -B -j $1 -f <(cat -) all

This uses a couple of clever tricks. I especially like the use of process substitution in the make command (substituting the 'cat -' for the input makefile).

This approach allows the commands in commands.txt to redirect their own output as they need to (using '>', '2>', '&>', etc.)

Bash: while [[ -e /proc/$process_id ]]

Sometimes you need to keep track of the amount of memory a process is taking up (or CPU, or something else). The /proc filesystem contains a subdirectory for each running process; the directory is named with the pid of the process. You can use this fact along with a basic file test and a while loop to track a process for as long as it lives.

> some_interesting_job.py &

> process_id=$(ps -o "%p %c" | grep "some_interesting_job" | cut -f 1 -d ' ');\
 while [[ -e /proc/$process_id ]];\
 do ps -o "%z";\
 sleep 5;\
 done

This will report on the virtual memory size (in KiB - see 'ps' manpage for more details) that the process is taking up. The while loop will terminate when the process completes (or is killed).

commandlinefu.com

I just rediscovered this excellent site. Well worth the occasional browse:

• Commandlinefu - main page
• Commandlinefu_byvote - sorted by vote count

There are some real gems in there! For example:

> python -m SimpleHTTPServer
Serve current directory tree at http://$HOSTNAME:8000/

Useful for tiding up your workspace whilst keeping jobs running:

> disown -a && exit
Close shell keeping all subprocess running

I do love a bit of process substitution:

> diff <(sort file1) <(sort file2)
diff two unsorted files without creating temporary files

Handy:

> rm !(*.foo|*.bar|*.baz)
Delete all files in a folder that don't match a certain file extension

And a "I should have thought of this; it's so obvious now!" trick:

> some_very_long_and_complex_command # label

Easy and fast access to often executed commands that are very long
and complex. When using reverse-i-search you have to type some 
part of the command that you want to retrieve. However, if the
command is very complex it might be difficult to recall the parts
that will uniquely identify this command. Using the above trick
it's possible to label your commands and access them easily by 
pressing ^R and typing the label (should be short and descriptive).

R: formatting numbers for output

I often produce tables of numbers with a lot of significant digits after the decimal point. It's confusing to look at 20 sig [fd]igs, especially in a large table of results, so I tend to format my output to make it more concise and easier to read.

The function 'format' is pretty good for this. Note that the 'format' call returns a character vector (which is fine if you're only going to write the number to file or the console).

Here's a simple example just using a randomly generated number:

> num <- rnorm(1, mean=10)
> num
[1] 10.24339

We call format with digits=4 (show 4 significant digits) and nsmall=4 (display at least 4 digits after the decimal - for real/complex numbers in non-scientific format):

> format(num, digits=4, nsmall=4)
[1] "10.2434"

You can see that the format command rounds the numbers. This uses the IEC 60559 standard - 'go to the even digit'. So 0.5 is rounded to 0 and 1.5 is rounded to 2...

Of course, if you're used to sprintf style commands then you can also use the sprintf function for this:

> sprintf("%.4f", num)
[1] "10.2434"

Simple parallel processing with xargs

We've all been there - looking in a nicely tidied up directory, full of archived data - hundreds of lovely data files all gzipped or (better) bzip2'ed; but now you want to use them and you have to uncompress them all... "if only I could use all n CPUs on my local machine to do this!": enter 'xargs'!

It's as simple as:

> ls *.bz2 | xargs -n 1 -P 6 bunzip2

This will set off bunzip2 on all bz2 files in the current directory.

The '-n 1' flag tells xargs to only provide one argument (file) per command line; the '-P 6' tells xargs how many concurrent processes to run.

hg over ssh

I'm working with a repo that I can't access using http, so it's just as well that Mercurial works just fine and dandy over ssh as well - you just have to do a little bit of tweaking to the basic configuration to get things working smoothly.

Add the following to the client .hgrc:

[ui]
remotecmd=<path_to_hg>

Where <path_to_hg> is the path to the hg executable on the main repo machine.

This was from StackOverflow: cloning-a-mercurial-repository-over-ssh

Now, I'm working very remotely from the main repo (6000 miles) so it is worth using compression. Just add the following to the [ui] section of .hgrc:

ssh = ssh -C

You could also configure ssh to use compression by default (see details in link below).

This page has a good overview of using ssh with mercurial: collaborating-with-other-people