GeekBrainDump

R: Empirical Cumulative Distribution Function

2011-12-06T17:45:00.001-08:00

There's a handy R built-in function for calculating the empirical cdf called ecdf. I had 40 separate distributions (molecular weight for different series of molecules) and wanted to look at the cdf for this. In R it's really straightforward. Here's an example where we simulate some molecular weight data and plot the CDF for the whole set:

# create the data
R> data <- data.frame(mw=rnorm(10000, mean=400, sd=50), name=1:10000, group=sample(c('a','b','c','d'), 1000, replace=T))
R> mw_cdf <- data.frame(cdf=ecdf(data$mw)(data$mw)*100, mw=data$mw)

R> qplot(mw, cdf, data=mw_cdf, geom="step", xlab="Molecular weight", ylab="Total (%)", main="CDF of MW")

The ecdf() call returns a function (of class "ecdf") which you can call with a MW to calculate the probability of seeing a molecule of this, or smaller, MW in your set.

The R code above produces the following plot:

I found this useful for looking at the MW distributions for disparate groups of molecules, both together and separately. If you're going to apply a molecular weight cutoff when analysing a set of molecules, it's good to get an idea of how many you'll be excluding and this provides a very quick way of seeing that information.

Running chrome on multiple displays

2011-12-01T16:50:00.001-08:00

By display here I mean $DISPLAY not multiple monitors.

I often leave Chrome running at work (on a ton of virtual desktops with a ton of tabs) and then get home and need to log into work (via VPN + NX) in order to check on things. It's frustrated me that if I try to run chrome in the NX session then it appears on my screen at work rather than in the NX session. Well, who would have thought that reading the man page would solve the frustration? This method sets up a new profile, so you loose all your bookmarks and settings, but that's ok for occasional use I think


/opt/google/chrome/google-chrome --user-data-dir=<dir>

Where <dir> is a dir of your choosing. The default for chrome is ~/.config/google-chrome/. So I've been using ~/.config/google-chrome/nx.Phew, friction resolved.

Using linux 'seq' with large numbers

2011-10-18T11:59:00.000-07:00

Sometimes I deal with quite a lot of data (IMO). Occasionally this has to be split into smaller files in order to be processed. I use 'seq' quite a lot for generating program lists to work on these files.

I ran into an issue the other day; I wanted to include the line-number offset in the filename of the files I was generating, unfortunately as soon as I hit a million lines the 'seq' numbers started to use scientific notation (i.e. rather than 1000000 seq output 1e+6) - unfortunately, this wasn't compatible with some of the downstream processing.

The 'seq' manpage seemed to claims that it accepts printf formatting arguments. So I tried running 'seq -f "%d" 0 10000 160000000' and 'seq -f "%i" 0 10000 160000000' but neither of these were recognised. It turns out that seq actually only recognises the printf style floating-point format... so to get it to work as desired you have to use "%.0f" instead:

> seq -f "%.0f" 1000000 1000000 10000000
1000000 
2000000 
3000000 
4000000 
5000000 
6000000 
7000000 
8000000 
9000000 
10000000

"Rule Engine" Knime node - matching missing values

2011-09-20T10:22:00.000-07:00

I just spent fifteen minutes trying to work out how to match missing values in the Knime "Rule Engine" node. It turns out you put the "MISSING" keyword before the column definition.

So, the rule looks something like this:

MISSING $species$ => 'blank'

R: invalid multibyte string

2011-05-05T09:13:00.000-07:00

I've just been using R for some string comparison work (pairing the titles from two reference sets where the formatting was quite different). I bashed my head against the wall for a little while trying to figure out why the analysis was failing with a "invalid mulibyte string 1" error.

Ok, the error itself is fairly obvious - the supposedly plain text strings contained some non-ASCII characters. Finding those characters was the issue. I tried grep ("[^:print:]" and a few others) and nedit but couldn't see the issue. Turns out just using less solved my problems. The non-ASCII characters are highlighted for all to see (so they're easy to remove using a text editor at that point).

Oh, and if you're interested I was using stringdot from the 'kernlab' package to compare the strings.

View a summary of filetypes in a dir

2011-04-06T11:13:00.000-07:00

Sometimes I fill up directories with lots of files of lots of types (different extensions). It's informative to be able to see what the distribution of types is in a dir. I tend to use this view during project cleanups to identify files that can be deleted or moved/consolidated. Anyway, here's a quick way of doing it:

for file in $(ls *.*); do base=$(basename $file); echo ${base##*\.};done | sort | uniq -c | sort -n -r
  37 tsv
  18 pdf
  17 txt
   7 err
   5 log
   5 png

Here I see that there are some log files that can be removed (if no longer needed).

Reading stdin from a pipe for command-line R shenanigans

2011-02-13T22:31:00.000-08:00

I think I rely on command-line tomfoolery a little bit too much. Today I wanted to pipe some data into R and have the commands to run defined on the command line as well. This is the kind of thing I do with Perl or bash to get some quick answers and I'd love to add R to the repertoire.

So, I tried a number of things all of which failed. This is how I got it to work for me:

> perl -le 'printf "%.4f\t%.4f\n", rand(), rand() for 1 .. 20' \
 | R --vanilla --slave -e\
 "data=read.delim(pipe('cat /dev/stdin'), header=F);\
  cor.test(data\$V1, data\$V2)"

You have to remember to escape any special characters in the R script ($ in this case).

R: using lattice graphs in functions

2011-01-12T14:51:00.000-08:00

I like the lattice graphics package; it produces nice looking and easy to generate graphs. However, I was bitten the other day when a function I'd written to generate a load of density plots wasn't creating the output files I was expecting. They just weren't there at all. After a little bit of unsuccessful poking around, I gave up and generated them manually (it was a stressful day with tight deadlines).

I went back to this issue today and it turns out that the lattice functions 'do not create plots'. You have to print the object created from the lattice function in order to see/output the graph.

This happens automatically on the command line, which is why I was so confused since the code would work fine outside of a function. If you want your graph to be outputted within a function you have to explicitly print it.

e.g.:

function write_density_plot(data, name)
{
    filename = paste(name, '_density_distribution.png', sep="")
    png(filename)
    print(densityplot(data, main=paste(name, 'score distribution'))
    dev.off()
}

filedata = read.delim('myfile', header=T)
write_density_plot(filedata$blorg, 'blorg')
write_density_plot(filedata$blarg, 'blarg')
write_density_plot(filedata$wib, 'wib')

EMC2, G64 and sharp corners

2010-11-28T00:04:00.000-08:00

I've cut a number of things on my CNC machine using EMC2. I've been perplexed by the fact that some of the corners in my test shapes were not sharp, but were rounded off.

I've just been designing and cutting some new leadnut holders and found that the hexagonal depression I was cutting was also plagued with these curved corners (not good when you need a hex nut to fit in them). It turns out this is to do with the G64 command in the G-Code which tells the trajectory planner to sacrifice path following accuracy in order to keep the feed rate up (i.e. cut/round corners). A good description is available in the EMC2 documentation section covering trajectory control.

I manually edited my G-code (this one coming from CamBam, but past ones were from Inkscape and dxf2gcode) so that the G64 command now read "G64 P0.001" and the hexagon cut with much sharper corners.

Fun with process substitution

2010-11-12T13:08:00.000-08:00

I was introduced to process substitution recently at work, it's a great way to avoid temporary files for some simple cases.

For example, say I wanted to know what the column header changes were between two files (same data, different code used to extract them). The files are tab delimited and I have a script I use (frequently) that prints out the index and name for the headers in an input file - columnHeaders.sh.

So, if I want to see what's different between two files, in the past I'd create two output files using my columnHeaders.sh script and then use diff, kompare or comm to compare them.

You can eliminate the temporary files using a technique called process substitution.

> diff <(columnHeaders.sh file1) <(columnHeaders.sh file2)
2,3c2,3
> 0 blarg
> 1 blorg
--
< 0 blorg
< 1 blarg

In this case we see two columns have been swapped between file1 and file2.

Take a look at the Advanced Bash Scripting Guide for more examples.

The columnHeaders.sh script basically does this (but allows user specified delimiters):

> head -1 <input file> | perl -F'\t' -lane 'print $n++,"\t$_" for @F'

Bash command line parameter parsing (getopts)

2010-09-17T15:49:00.000-07:00

If I only need a single optional parameter I'll often just check to see if the input positional parameter of interest is set (e.g. script that accepts one required field and one optional field):


if [[ ! -n "$2" ]];
then
    echo "I received the second parameter:$2"
fi

But if you want to do something a bit more complex, getopts is your friend.

For example, say you want to have the user input their first name, last name and a "keep my data private" flag you could do something like this:


while getopts f:l:p flag
do
    case $flag in
        f)
            FIRSTNAME=$OPTARG;;
        l)
            LASTNAME=$OPTARG;;
        p)
            PRIVATE=1;;
        ?)
            echo "$0 -f <first name> -l <last name> -p"
            echo -e "\t-p [flag] keep my data private"
            exit
done

The getopts command is fairly straightforward (man getopts for more details). If an option requires an argument then a colon is placed after it's letter designation ('f' and 'l' in the above example).

You can check for required parameters by looking at which variables were set:


if [[-z "$FIRSTNAME" || -z "$LASTNAME" ]];
then
    echo "missing required parameter"
fi

Wrap that all up into a neat script with a subroutine that outputs a usage statement and you're home free:


function usage_and_exit()
{
    echo "$0 -f <first name> -l <last name> -p"
    echo -e "\t-p [flag] keep my data private"
    exit
}

while getopts f:l:p flag
do
    case $flag in
        f)
            FIRSTNAME=$OPTARG;;
        l)
            LASTNAME=$OPTARG;;
        p)
            PRIVATE=1;;
        ?)
            usage_and_exit;;
done

if [[ -z "$FIRSTNAME" || -z "$LASTNAME" ]];
then
    echo "missing a required parameter (firstname and lastname are required)"
    usage_and_exit
fi

if [[ $PRIVATE -ne 0 ]];
then
    echo "protecting private data"
fi

MySQL: the mystery of unsetable global variables

2010-08-18T11:41:00.000-07:00

I just tried updating our mysql server to accept very long connections. I have a ton of jobs running, so I wanted to set the wait_timeout variable (via the mysql shell) to something reasonable for these jobs. The default of 8 hours is not sufficient in some rare cases so I tried to set the timeout higher:


mysql> SHOW VARIABLES LIKE 'wait_timeout';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| wait_timeout             | 28800 |
+--------------------------+-------+
1 row in set (0.00 sec)

mysql> SET GLOBAL wait_timeout=86400;
mysql> SHOW VARIABLES LIKE 'wait_timeout';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| wait_timeout             | 28800 |
+--------------------------+-------+
1 row in set (0.00 sec)

What? Why wasn't it set?

Well, the reason is that the SHOW VARIABLES command defaults to the session variables. So, the local session wait_timeout is still 2880, but the global wait_timeout was actually updated correctly:


mysql> SHOW GLOBAL VARIABLES LIKE 'wait_timeout';
+--------------------------+-------+
| Variable_name            | Value |
+--------------------------+-------+
| wait_timeout             | 86400 |
+--------------------------+-------+
1 row in set (0.00 sec)

Basic MySQL 'top' command

2010-07-14T17:15:00.000-07:00

Ever wanted to keep an eye on the running processes in a MySQL database? How about something that works a little like the top command, but without any of the bells and whistles?

Well, here you go:


> while [[ 1 ]];
do
 clear;
 mysql -u <username> -p<password> -h <host> -e "show full processlist";
 sleep 1;
done

Remember to replace the <username> <password> and <host> variables with the values for your database. Also, if you don't like the bounding box on the mysql output, you can have a cleaner output by using redirection instead of the -e flag:


mysql -u <username> -p<password> -h <host> <<< "show full processlist";

Case insensitive regex in bash

2010-06-16T10:49:00.000-07:00

By default regexes in [[ ]] are case sensitive. If you want to match in a case insensitive way you have to set the shell option nocasematch.


mytext="eXistenZ"
if [[ $mytext =~ existenz ]];
then 
 echo "yep"
else
 echo "nope"
fi

If you run the above script you should get "nope" as the output. For case insensitive matching just insert this into the script prior to the regex:


shopt -s nocasematch;

You can unset the nocasematch shell option using the following: shopt -u nocasematch

Here's a more complete example:


mytext="eXistenZ"
function testText
{
 if [[ $mytext =~ existenz ]];
 then 
  echo "yep"
 else
  echo "nope"
 fi;
}
testText
shopt -s nocasematch
testText

If you run that script then the output is:


nope
yep

R related links

2010-06-13T22:38:00.000-07:00

These are just a few R links I think are interesting and worth a further read:

Colour palettes in R.

Using R for introductory statistics

Web-friendly visualisations in R

A different way to view probability densities

Thoughts on Making Data Work

Also of note is Google's announcement (on June 2nd) that they were working with the USPTO to make all granted patents, trademarks and published applications freely available for download - see this post.

I love subshells

2010-06-01T09:35:00.000-07:00

Sometimes it's the little things in life that give us the greatest pleasures. I love subshells. There, I admitted it.

Say you want to do something with the output of a program but would like to prepend some output/text that also needs to be operated on. The standard idiom would be to create a file with the prepended output, append your output to this and then operate on the file, but you can remove the intermediate files using a subshell. Here's a contrived example:


> (echo -e "some\toutput\tto\tappend"; perl -lane 'print join("\t", log($F[0])/log(10), @F[1 .. #$F])' ;) | a2ps

Something else I use subshells for a lot is to launch programs in a different directory without having to cd there and then back again:


> (cd ~/workspace/; eclipse &)

like I said; it's the little things.

delete all empty files in a directory

2010-05-18T16:28:00.000-07:00

Automatically generating files? Annoyed at all the empty ones? Here's how to purge them:


> for file in $( ls . ); do if [ ! -s $file ]; then rm -f $file; fi; done

Of course you could just get over your fear of find and take a look at that man page. Perfect for a recursive search and delete all in one:


> find . -empty -delete

simple, no?

escaping quotes in bash variables

2010-05-14T16:08:00.001-07:00

I often need to escape quotation characters or other special characters that are being piped into a bash process:

Say you have a mysql table which contains protein names. Some of these have the ' character in them (e.g. "Inosine 5' monophosphate dehydrogenase") and you want to do some bulk processing on these names, you could do something like this:


> mysql -u paulbo -N <<< "SELECT name from proteins" | while read protein_name; do mysql -u paulbo -N "SELECT count(*) FROM data INNER JOIN proteins ON data.protein_id = proteins.protein_id where proteins.name = '${protein_name//\'/\\\'}'"; done

Ah, three backslashes. Why didn't I think of that?

And sometimes you just want to output the text with the special characters all converted into something a bit more amenable (like the old and trusted '_').


> mysql -u paulbo -N <<< "SELECT name from proteins" | while read protein_name; do echo ${protein_name//[-\/\:\'\"\(\) ]/_};done

Excluding certain files from a directory listing

2010-05-06T13:40:00.000-07:00

Ok, well, this will be obvious to anyone who's really read the ls man page, but I only came across it a couple of days ago.

Say you have a directory with tons of files in it mostly with a single extension and you want to see what else is in there. Sure, you can use grep, but you can also use ls's inbuilt flat --hide.


> ls -l
arg.txt
blarg.txt
foo.txt
...
interesting.sh
... # and a ton more *.txt files
zlargyblorg.txt
> ls -l --hide=*.txt
interesting.sh
hidden.csv

Isn't that useful?

batch renaming files with spaces

2010-04-30T17:48:00.000-07:00

Why do people keep giving me tons of files with spaces in the filename?

Anyway, here's a good way to get rid of those pesky spaces:


ls * | while read file; do mv "$file" ${file// /_}; done

First I tried using "for file in `ls *`" but of course the whitespaces came back to bite me... This was also true for the mv command. You have to quote "$file" in order for the whitespace ridden filename to be recognised as a unit rather than multiple file descriptors.

bash array size

2010-04-27T17:11:00.000-07:00

Strangely this caught me off guard. If you use ${#array[@]} to get the 'size' of an array, it actually only returns the number of assigned elements in the array.

e.g.


> array[23]=123
> echo ${#array[@]}
1

Hmm... only 1? Not 24?

As far as I can tell, there's no way around this. Just don't expect this behaviour and fill your arrays wisely.

While we're on bash arrays, remember that you can change the 'join' character for naive printing of arrays by manipulating the IFS (Internal Field Separator) variable. Below I also show that the quotation context is important for this:


> array[0]=1;array[1]=2;array[2]=3;
> echo ${array[*]}
1 2 3
> echo "${array[*]}"
1 2 3
> IFS=","
> echo ${array[*]}
1 2 3
> echo "${array[*]}"
1,2,3
>

Note: It's best practice to store and then restore the original IFS variable.


> ORIG_IFS=IFS
... do stuff
> IFS=ORIG_IFS

Java: Heap Dump on OutOfMemoryError

2010-04-15T16:09:00.001-07:00

You can request the jvm create a heap dump when an OutOfMemoryError is thrown. This is handy if you have a process that consumes a ton of RAM and you don't know why. Set the max heap size to something around 500M (or less. It needs to be fairly small if you're going to inspect the heap with 'jhat'). Use the -XX:+HeapDumpOnOutOfMemoryError flag to request the heap dump. This will output to java_pid.hprof by default. You can set the output filename manually using -XX:HeapDumpPath=<filename>.

e.g.


> java -Xmx100m -XX:+HeapDumpOnOutOfMemoryError -XX:HeadDumpPath=/tmp/dump.hprof com.geekbraindump.MyMemoryHoggingClass

Linux: command line cut and paste

2010-03-23T10:07:00.000-07:00

It's always annoyed me that I have to open up a file in order to cut and paste the contents into a web browser (I use this a lot for capturing information on an internal wiki). As of 2010-03-23 there are no default command line access utilities for this (on CentOs anyway).

However, download and install xclip and the clipboard is yours to command.

xclip allows access to both the PRIMARY (middle mouse button) and SECONDARY (standard copy/paste) selections.

By default piping into xclip puts the text in the PRIMARY clipboard (middle mousebutton).


> echo $RANDOM | xclip
> xclip -o
4807

You can define which selection to input to. Say you want to store text in the SECONDARY selection (accessed using standard cut and paste commands):


> echo $RANDOM | xclip -sel 'clipboard'
> xclip -o -sel 'clipboard'
4807

You can now use edit->paste to output the text. Note that the random number was the same as before. I guess this is some shell caching mechanism. For a new random number each time you have to use a new shell:


> (echo $RANDOM) | cat
1297

AVR: storing a 2d array in PROGMEM

2010-02-28T23:58:00.000-08:00

I've just been fangling with a UV Painter project (row of LEDs to 'paint' on a glow-in-the-dark-wall). I quickly ran out of RAM when adding patterns to the system and learned how to add variables to the Flash program memory instead. Pretty simple and very useful:


#include <avr/pgmspace.h>

const uint8_t mCylonScan[10][N_LED] PROGMEM = {
  {255,0,0,0,0,0},
  {0,255,0,0,0,0},
  {0,0,255,0,0,0},
  {0,0,0,255,0,0},
  {0,0,0,0,255,0},
  {0,0,0,0,0,255},
  {0,0,0,0,255,0},
  {0,0,0,255,0,0},
  {0,0,255,0,0,0},
  {0,255,0,0,0,0}
};

Then to access the data you just do the following (where 'i' and 'j' are loop variables):


data = pgm_read_byte(&(mCylonScan[i][j]));

See the AVR libc docs for more details.

Manipulating Bash Strings

2010-02-26T15:34:00.000-08:00

I'm finding myself looking these up quite a lot, so here's a little cheat sheet of the basics.

Using the string "ABCDEFG12345" as an example:


> string="ABCDEFG12345"
> echo $string
ABCDEFG12345

> #Replacement:
> echo ${string/ABC/___}
___DEFG12345

> #Replacement with character class
> echo ${string/[ABC4]/_}
_BCDEFG12345

> #Replace all occurrences:
> echo ${string//[ABC4]/_}
___DEFG123_5

> #Extract from a defined position in the string
> echo ${string:7}
12345
> echo ${string:7:3}
123

> #substring removal (from the front of the string)
> echo ${string#ABC}
DEFG12345
> echo ${string##ABC} #strips the longest substring match
DEFG12345
> string2="abcABCabc123ABCabc
> echo ${string2#a*C}
abc123ABCabc
> echo ${string2##a*C}
abc
> # use the % sign to match from the end 
> # % for shortest and %% for longest substring match
> echo ${string%45}
ABCDEFG123

Use csplit to split SDF files (or contextually split any file)

2010-02-11T10:52:00.000-08:00

Say you want to split an SDF into individual entities, you could write a Perl script/one-liner (which is what I've been doing for a long time) or you could just use csplit. Thanks to Pat and Jessen for pointing this one out.

e.g. say you had an SDF, test_mols.sdf, with 8 molecules in it and you wanted individual mol files:

> csplit -kzsf "test_mols" -b %0d.mol test_mols.sdf /\$\$\$\$/+1 {*}

This would result in 8 files called test_mols00.mol through test_mols07.mol. Unfortunately these would still contain the SDF delimiter at the end of the file (so, technically these are still SDFs). That's pretty easy to clean up with something like:

> perl -ni -e 'print unless /\$\$\$\$/' *.mol

See the csplit manpage for more details.

autolinkification in bugzilla

2010-02-02T14:31:00.001-08:00

I often refer to other bugs in a Bugzilla comment. These are 'autolinkified' by bugzilla. I've only just learned that you can refer to a comment in a bug as well and have this 'autolinkified'.

See the Bugzilla hintandtips page.

Short answer:

Bug autolink:" bug 1234"
Comment autolink:"bug 1234, comment 12"
attachment: autolink" bug 1234, attachment 4"

Regular expressions in bash

2010-01-29T11:39:00.000-08:00

You can perform regular expression matching on a variable within an extended test command (see the Conditional Constructs part of the bash manual).

e.g.


prompt> name=foobar.blarg;if [[ $name =~ foo ]]; echo yep; else echo nope; fi
yep
prompt> name=foobar.blarg;if [[ $name =~ foo[a-c] ]]; echo yep; else echo nope; fi
yep
prompt> name=foobar.blarg;if [[ $name =~ foo[d-z] ]]; echo yep; else echo nope; fi
nope

Convert tab data into HTML tables with a Perl one-liner

2010-01-26T16:04:00.001-08:00

Quick one-liner for generating a HTML table from tab delimited input. Either pipe in your data or include the file as a command line argument.


perl -F'\t' -lane 'BEGIN{print "<table border=1 cellpadding=3 cellspacing=0>"}print "<tr>", (map {"<td>$_</td>"} @F), "</tr>";END{print "</table>}'

The map is in parentheses so that the closing '<tr>' tag is not slurped in as part of it's input array.

Setting AVR Clock Speed

2010-01-16T09:51:00.000-08:00

I'm using gcc/WinAVR to program my AVRs and have just been discovering how to program the clock speed. I'm playing with an ATtiny13 and an ATmega8. Both of these ship with their clocks set to 1MHz by default but both can be clocked to a higher speed.

There are a few things to note:

1. F_CPU is used by the compiler for calculating timings (most obvious example is in the delay.h routines). Setting it has no effect on the actual clock speed. This needs to be set to the correct value.

2. Easiest way (for me) to set the clock speed is to program the relevant fuse bits. This was different for the two chips I've been using - ATtiny13 and ATmega8). Note: for fuse bits 1 = unprogrammed and 0 = programmed (this is due to the nature of EEPROM).

This step is made easy when using the Eclipse avr plugin. There's a GUI/wizard for setting them in Project->Properties->AVR->AVRDude->Fuses. Accessible by selecting the "direct hex values" radio option and then clicking the "start editor" button.

3. For some MCUs you can dynamically adjust the clock speed in software (I know this is true for the ATtiny13 at least). However, this has to be done within 4 clock cycles of setting the CLKPCE bit (again, this is for the Atiny13). See this forum post on avrfreaks.net and pg.28 of the datasheet.

Here's a good overview of setting the clock for the ATmega8: Electrons - AVR Fuses HOWTO Guide.

Bash: reading lines from a file

2010-01-12T14:36:00.000-08:00

I guess I don't do this enough to remember it:

When reading a file in a for loop in bash, the following idiom will read each work (whitespace delimited):

for word in `cat file.txt`; do echo $word; done

If you want to grab the whole line then you can do this:

while read line; do echo $line; done < file.txt

cat file.txt | while read line; do echo $line; done

Essential Eclipse Keyboard Shortcuts (navigation)

2010-01-08T10:48:00.001-08:00

There are a few shortcuts I use all the time for navigating in Eclipse and I've just learned a few more useful ones (I was looking for quick ways to jump between editor windows).

Here's the essential list (IMO):

Ctrl+E (go to other open editors - opens selection box)
Ctrl+Q (jump to last edit location)
Crtl+O (jump to any member/method/inner-class in the current editor)
Ctrl+shift+T (open any type)
Ctrl+shift+R (open any file)
Ctrl+L (jump to a particular line number)
Ctrl+T (go to a supertype/subtype - multiple presses toggle between super/sub)
Alt+left/right arrow (jump through visited files)
Ctrl+. Ctrl+, (navigate up and down through error/warning locations)

C rand() and random() functions

2010-01-03T22:34:00.000-08:00

There are two primary random functions to be aware of in stdlib.h: random() and rand() the main difference is in the range of values returned by the two functions.

random() returns a pseudo random number in the range 0 -> 0x7FFFFFFF = 0 -> 1,879,098,192 (RANDOM_MAX).
rand() returns values in the range 0 -> 0x7FFF = 0 -> 28,672 (RAND_MAX).

I had mistakenly been using one in place of the other in some micro-controller code and had spent some time wondering why it wasn't behaving as expected...

updating windows vista file permissions with Icacls

2009-12-24T10:26:00.001-08:00

I'm not much of a windows guy, but I have a vista machine at home as the main PC. I recently bought a NAS to backup important files and found that my backup jobs were failing due to file permission issues (there are multiple accounts on the PC). I wanted to do the equivalent of chmod +r (or even 777) on a few files but didn't want the hassle of using windows explorer to adjust the file perms one at a time (I'd already tried changing permissions for the root folder and applying these to the contained files and sub-dirs, but it didn't work. I guess it was something to do with the fact that the permissions weren't uniform for the files in the dirs).

Anyway, it seems you use Icacls for updating file permissions and you can do it recursively with the /T flag. So for a single file you do this:


> Icacls <filename> /grant <user>:<perm>

e.g. This grants full control to user 'Paul' on file 'test.txt'


c:\> Icacls test.txt /grant Paul:F

and for a dir you do this:


c:\Icacls <dirname> /T /grant <user>:<perm>

e.g.


c:\> Icacls c:\testdir /grant Paul:F

Mysql multiple counts on one line (reports)

2009-12-21T11:11:00.001-08:00

I like to poll the contents of multiple tables over time in order to track the progress of certain processing tasks I do. I used to perform individual counts for each table and kept doing that out of habit. There's a much nicer way to do this and have the report in a single result. I tend to keep track of the elapsed time using the unix_timestamp() function (this returns the number of elapsed seconds since Jan 1st 1970 as an unsigned integer).

First set up the initial variables (those to compare against - this is time 0)


PaulBo@test_db:SELECT @time:=UNIX_TIMESTAMP(), @apples:=(SELECT COUNT(*) FROM apples), @oranges:=(SELECT COUNT(*) FROM oranges);

Then, let some time pass and poll the tables for changes:


PaulBo@test_db:SELECT UNIX_TIMESTAMP() - @time as elapsed_time, (SELECT COUNT(*) FROM apples) - @apples as d_apples, (SELECT COUNT(*) FROM oranges) - @oranges as d_oranges;
+--------------+----------+-----------+
| elapsed_time | d_apples | d_oranges |
+--------------+----------+-----------+
|          435 |      230 |     12887 |
+--------------+----------+-----------+
1 row in 1 set (0.00 sec)

Redirecting the output from a "here document"

2009-12-11T11:23:00.000-08:00

This had me confused for a few minutes, so I thought I'd post.

It's common to use "here documents" to simplify input to a program in a script.

e.g.


> cat <<EOF
> This is a random number:
> $RANDOM
> EOF
This is a random number:
9948

But what if you want to capture the output of this? The naive attempt would be to redirect after the second EOF, but this is incorrect as the termination string has to be on a line all by itself.

This is the answer:


> cat <<EOF > /tmp/data.txt
> This is a random number:
> $RANDOM
> EOF
> cat /tmp/data.txt
This is a random number:
24669

samples from large datasets in R

2009-12-11T10:22:00.000-08:00

I have a dataset I want to plot (say 5,000,000 data-points). This can be very slow to plot in R, so you want to take a sample of this data instead.

Say I have a tab delimited file with two columns, say 'time' and 'count'. The columns have these as headers. There are 5M rows and I'd like a simple overview of the count over time.


> data = read.delim('filename', header=T) #read in the tsv (tab separated value) data file
> s = length(data$time) # calculate the number of data-points
> n = 1000 # this is my sample size
> N = sort(sample(1:s, n)) #create a set of indices sampled from the vector 1:s
> plot(data$time[N], data$count[N]) # use the indices to sample from the set

The magic is in the sort(sample(1:s, n)). This takes n samples from the space 1 -> s. Unless a probability vector is provided, each element in the input vector (1:s) has an equal probability of being selected. We sort the output of sample so that the indices are in the correct order to plot. Actually I just tried this without the sort and it seems the plot() function sorts the input vectors anyway.

Infinite loop in bash

2009-12-03T15:21:00.000-08:00

I needed an infinite loop for polling a database table for a while (killed with Ctrl-C).


while [ 1 ];
 do mysql -u paul -h test_database -ppass test_paul data_table <<< "show processlist";
 sleep 10;
done

Visual Diff

2009-11-20T09:15:00.001-08:00

I just came across 'kompare'. It's a great way to track minor differences between plain text files. Very pretty, side-by-side, visualization of the changes.

The display uses 3 colours:

Green - new text on the left
Blue - new text on the right
Red - changed text/area

See the wikipedia page for more details.

Using curl to POST data

2009-10-23T11:35:00.000-07:00

I wanted to automate some load testing at work, this required sending some post requests to a running server in order to increase the amount of 'work' being done. I wanted to use 'curl' as it's a linux command line app and so very easy to incorporate in a script.

Anyway, this is how you post data with curl:


curl -d "param1=value1¶m2=value2" http://myhost.com/server.cgi

Inserting binary values in the mysql shell

2009-10-02T10:54:00.001-07:00

I needed to insert some dummy values into a table which had a bit field as well as a blob field, both of which are "NOT NULL". You just need to have the 'b' prefix on the data, like so:


INSERT INTO faketable(blob1, name, bitfield) values(b'001011101', "Beeblebrox", b'1');

You can also print out the binary data in a way that's not going to ruin your terminal using the BIN(), OCT() and HEX() functions:


SELECT name, HEX(blob1), BIN(bitfield) from faketable;
+--------------------+--------------+----------------+
| name               | HEX(blob1) | BIN(bitfield) |
+--------------------+--------------+----------------+
| Beeblebrox         | 005D         | 1              | 
+--------------------+--------------+----------------+

R - percentiles

2009-09-14T17:03:00.001-07:00

I just needed to find some percentile thresholds and thought that it'd be easy with R. Well, it is!

So, load the data


data = seq(0,1,0.01);

Then you can get quantiles like so:


quantile(data);

   0%  25%  50%  75% 100%
0.00 0.25 0.50 0.75 1.00

and percentiles by using the vector argument to quantile (in this case I'm looking at the 0th to 1st percentile):


   0%  0.1%  0.2%  0.3%  0.4%  0.5%  0.6%  0.7%  0.8%  0.9%    1%
0.000 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010

screen

2009-09-11T09:13:00.000-07:00

Ever lost a terminal which had an active job running? Ever wanted to check in on what you'd been doing from home *and* wanted or needed the session history as well?

'screen' is your friend. If you're starting work that's important or that you may want to check from another machine it's a real time-saver. Have a look at the man page for more details than you can shake a stick at.

I've only been using it for a little while, so the only caveat I've found so far is that I have to remember to give the sessions a reasonable name before I start them. Otherwise, if you've got a lot of sessions, it's a time waster trying to find the right one. The '-t' flag sets the title for the default shell; more importantly the '-S' flag sets the session title. I've been using the same string for both (will probably set up an alias to deal with the duplication soon).

e.g.


shell1> screen -t data_loading -S data_loading
shell2> screen -ls
There are screens on:
        21802.data_loading    (Attached)
        20237.pts-10.hamilton   (Attached)
        1376.variance_analysis  (Attached)
        6639.pts-8.hamilton     (Attached)
        9881.cluster_jobs        (Attached)
5 Sockets in /var/run/screen/S-paul.

You can see from the above listing that there are 5 sessions (all currently attached). There are two listings there with no session title and I have no idea what I'm doing in them... but for the other three it's obvious (at least to me).

Code formatting for blogger

2009-08-31T11:10:00.001-07:00

I've been meaning to look into this for a while. I'd like to be able to insert nicely formatted code into posts.

I found the follwing blog post - getting-code-formatting-with-syntax-highlighting-to-work-on-blogger and figured I'd give it a try.

Unfortunately following the instructions in the above post didn't work for me and I had to waste some time on the project site to find out what to do. This post on puffyandmishu was very helpful. I ended up with the following code in my blogger template:

1. In the <head> tag:


<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shCore.js'/>  
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushCpp.js'/>
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushBash.js'/>  
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushJava.js'/>
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushSql.js'/>
<script language='javascript' src='http://alexgorbatchev.com/pub/sh/current/scripts/shBrushPerl.js'/>
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shCore.css' rel='stylesheet' type='text/css'/>
<link href='http://alexgorbatchev.com/pub/sh/current/styles/shThemeDefault.css' rel='stylesheet' type='text/css'/>

2. at the bottom of the <body> tag:


<script language='javascript'>  
SyntaxHighlighter.config.bloggerMode = true;
SyntaxHighlighter.all()  
</script>

Let's test it out:

C - <pre class="brush:c">


int main(void)
{
    uint8_t i = 0;
    char c = 'a';
    while(1)
    {
        i++;
    }
}

Bash - <pre class="brush:bash">


for i in `ls *.c`;
  do echo $i;
done

Sql - <pre class="brush:sql">


SELECT name FROM mydb.test_table WHERE id IN (1,2,3,4,5);
INSERT INTO mydb.test_table VALUES(1) WHERE name = "test";

Java - <pre class="brush:java">


private static final class MyClass extends YourClass implements Cloneable
{
    public int mMyInt = 145;
    private String mString = "blarg";
    protected double mDouble = 23.45;

    public void main(String[] pArgs) throws Exception
    {
        System.out.println("Hi there");
    }
}

Perl - <pre class="brush:perl">


use warnings;
use strict;

my $i = 10;
for my $j (0 .. $i)
{
    print "This is line $j.  How lovely!\n";
}
warn "this does nothing.";

Line wrapping off <pre class="brush: plain; wrap-lines: false">


This is an example of a long line that shouldn't wrap.  Dwi eisio ddweud rhiwbeth yn Cymraeg ond dwi wedi anghofio popeth.  How long can we make these bars?

Now lets try it with Perl.


perl -F'\t' -lane 'print if $h{$F[0]}++;END{print "$_\t$h{$_}" for keys %h}' averyververververlongfilename.tsv

AVR delay_ms

2009-08-21T23:43:00.001-07:00

When programming the AVR uC you have to remember that the delay_ms and delay_us functions will tie in the whole floating point library unless they are passed constants. This will always make the code too bloated to fit in anything less that 3K. So, it's right out for the ATtiny13 (1K Flash memory). I've been bitten by this twice now and the incidences were far enough appart that it took me 45 mins the second time to trace what was going on...

I was trying this:


delay_ms(delay);
if(i==0)
delay--;

but should have been increasing/decreasing the delay time by wrapping the delay call in a loop of variable length instead.

Legends in R plots

2009-08-21T16:20:00.000-07:00

I've just spent about 45 mins trying to get a decent results plot from R. I wanted to include a legend for clarity and this gave me some trouble. It turns out the co-ordinate system was using values from the x and y plot rather than pixels... weird. Happily, for simple plots, there's an easy way to position the legend. From the R man page:

The location may also be specified by setting 'x' to a single
keyword from the list '"bottomright"', '"bottom"', '"bottomleft"',
'"left"', '"topleft"', '"top"', '"topright"', '"right"' and
'"center"'. This places the legend on the inside of the plot frame
at the given location.

So, I ended up with something like this:

R --vanilla --slave <<<"d=read.table('data1.csv', sep=',', header=F);e=read.table('data2.csv', header=F, sep=',');png('results.png');dh=hist(d\$V2, plot=F);eh=hist(e\$V2, plot=F);xlim=range(dh\$mids, eh\$mids);ylim=range(dh\$counts, eh\$counts);plot(dh, main='', xlab='value', col='orange', xlim=xlim, ylim=ylim);plot(eh, col='lightblue', add=T);legend('topright', legend=c('label1', 'label2'), fill=c('orange', 'lightblue'));q()"

Pre-calculating the range for the x and y axes (using the range operators) prevents you from cutting off data from one of the plots.

MySQL bash shell one-liners

2009-08-06T14:15:00.001-07:00

This is a useful thing to know if you're ever trying to use data in files to grab data from a MySQL database: You can feed in SQL query strings to the mysql client in the bash/shell standard way:


bash> mysql -u paul -h myhost -pmypass test_paul <<< "SELECT id from a_table where a_name='boo-yah';"

Which means that you can put this in a shell script (or Perl script, or whatever). For example, to print out the addresses associated with a set of names:


for name in `cat names.list`;
   do address=`mysql -u paul -h myhost -pmypass test_paul <<< "SELECT address from a_table where a_name='$name';"`;     echo "$name -> $address";
   done

Perl subroutine references

2009-07-29T15:02:00.001-07:00

I just found this useful for generating a dispatch/lookup table. You can create a hash table with the values being references to subroutines.

Here's a simple example:


> perl -le '$a = sub { 5 * $_[0]};print &{$a}(23);'
> 115

A better example is to use this as a look-up table. Here's another simple example:


%table = (
 "+" => sub { $_[0] + $_[1] },
 "-" => sub { $_[0] - $_[1] },
 "*" => sub { $_[0] * $_[1] },
 "/" => sub { $_[0] / $_[1] },
);

print &{$table{"+"}}(12,24), "\n";
print &{$table{"-"}}(24,12)), "\n";

This would output 36 and 12 respectively. This is very handy for more complex processing and parsing (which is what I'm using it for).

Command line R

2009-07-21T10:09:00.000-07:00

Sometimes I just want to print out a histogram from a file, or create a simple summary of some numeric data. It's good to be able to just bash off an R script from the command line to do this for you.

There are a couple of easy ways to invoke R to run in 'batch' mode (i.e. non-interactive):
1. R CMD BATCH <scriptname>
2. R --vanilla --slave <scriptname>

For 1. the output is saved in a file named <scriptname>.Rout, in 2. the output comes to STDOUT - which, for me, is much more useful.

Also, you can use the standard shell 'tricks' to create quick scripts without having to save a script file:

e.g.

1. Create a numeric summary of the input data (in this case for a file with the format - "name,value"):


R --vanilla --slave <<< "d=read.table('data.scores', sep=',');summary(d);q()"
       V1       
 Min.   :1.333  
 1st Qu.:4.037  
 Median :4.651  
 Mean   :4.634  
 3rd Qu.:5.282  
 Max.   :8.000

2. Create a histogram for the data and save to a PNG (don't forget to escape any special shell characters).


 R --vanilla --slave <<< "d=read.table('data.scores', sep=',');png('data.png');hist(d\$V1);q()"

2009-07-16T16:14:00.001-07:00

Never tried SMS blogging before... Worse than Twitter?

converting postscript to an image

2009-07-15T18:06:00.000-07:00

This is something I've meant to learn how to do for ages. I needed to do this to produce summaries for a shit load of PDFs (containing related molecules) and wanted to have an image of a structure to represent each PDF. Anyway, I print structures as postscript files, so I needed a way to create a PNG of the first structure for each PDF.

I saw a few solutions on line which used ghostview to first output a jpeg and then 'mogrify' to cut down the hunormous result to something useful. It turns out you can just do this:

convert molecule.ps -trim molecule.png

This creates a png from the ps file. Blimey! Really simple!

The -trim tag is a great addition as well, since a direct conversion results in a lot of extra whitespace.

running command line perl within a bash script

2009-07-15T14:13:00.001-07:00

Maybe it's just me, but this is something I've struggled with on a couple of occasions.

Since I can't give any of the examples I'm actually working on, I'll have to use something which is less obviously useful.

The simple version is to use bash to loop through some files and use Perl to print out the filename (as stored by the bash script):


for file in `ls *.label`;
   do perl -le "\$a=1;print \"${file} \$a\"";
done

The important things to make this work are:
1. The double quotes used for the Perl script. This enables the variable interpretation by bash.
2. Escape sequences for the double quotes used in the Perl script - the double quotes are necessary to make Perl interpret the Perl variables.
3. Escape sequences for the Perl variables - this distinguishes the Perl variables from the bash ones.

The final script I used was a lot more complicated than this and was used to generate a series of files. I guess I can put the script in as, with no context, it has little meaning.


for si in `seq 38 47`;
 do perl -F',' -lane "if(@F==4)
   {
      print \"\$F[0]\\t\$h{\$F[0]}:\$F[1],\$F[2]\"
        if \$F[3] == $si and \!\$i{\$F[0]}++;
   }
   else{
      @F=split/\t/;
      \$h{\$F[1]} = \$F[0];
   }" file1 file2 > output_${si}.tsv;
done

It would have been nicer just to write a separate Perl script and call that, really. But there you go.

using find to catch up on what's changed

2009-06-18T10:28:00.000-07:00

I came back from a two week vacation and needed to find out what had been updated/added/changed in that time in some shared project dirs. "find" was my friend. The following command gives a list of all files changed within the last two weeks.

find . -ctime -14

'-ctime -14' will be true for all files/dirs that have been changed in the last 14*24 hours. You can alter the behaviour by changing the modifier on the numerical input:


+n     for greater than n,
-n     for less than n,
 n     for exactly n.

jvisualvm instead of kill -3

2009-05-18T15:20:00.000-07:00

Whenever I need to see what's going on under the hood of a process I issue a kill -3 to get a stack-trace. Well, I was just introduced to jvisualvm! This is a great way to see what's going on with a running java process.

Changing the mysql prompt

2009-05-18T14:56:00.000-07:00

This is such an obvious thing to do, why didn't I do it before?

I'm constantly jumping into and between multiple databases as multiple users (test, standard, mysql) it becomes a little dangerous and I'm finding myself checking where I am and who I am:
SELECT user();
SELECT database();

but you can change the standard mysql prompt to display the user and database! Just edit the my.cnf file and place the following under [mysql]:

prompt=\\u@\\d:

Now my prompt looks like this:
paul@test:
or
root@main:

If you don't have permissions to alter the my.cnf directly, then you could change your personal .bashrc (or equivalent) to contain something like this:

alias mysql='mysql --prompt="\u@\d:"'

This has saved me a lot of paranoia (and probably some headaches).

bash substring substitution

2009-05-01T21:26:00.000-07:00

I just came across this today, it's something that I wish I'd known years ago (I think it just shows that I need to read more manuals), I've been emulating this feature with Perl and regexes or with the basename/dirname commands.

It's very simple; today I wanted to run through a set of files with a particular file extension, do some processing and then print the output to another file named the same as the original but with a different extension.

So, say the files are .tsv (tab delimited) and I want to create a .csv (comma delimited) one from the first two columns, you can do it like this:


for file in `ls *.tsv`;
  do cut -f1,2 --output-delimiter=, \
  $file > ${file/.tsv/.csv};
done

Sweet, eh?

Histograms in gnuplot

2009-04-24T13:06:00.000-07:00

I'm pretty new to gnuplot, but have found it to be a handy tool for simple plots.

Today I needed to generate a simple histogram. Rather than creating a permanent script I've just been piping a string into gnuplot from echo.

echo "set terminal png;\
 set output 'timings.png';\
 set style fill solid 0.5 border -1;\
 set xlabel 'chunks';\
 set ylabel 'time to complete (m)';\
 set key autotitle columnhead;\
 set title 'timings on 20 nodes';\
 set auto x;\
 plot 'timings.tsv' u 2:xticlabels(1) with boxes lt 2;"\
 |  gnuplot

and out pops a nice histogram (sorry, can't post this example as it's work related).

The input file had column headers and the first column was used as the xlabels (see the xticlabels call above?)

Stacktrace from a running process

2009-04-23T10:07:00.001-07:00

There have been a few occasions where I've been very confused as to why a process is taking a long time to run (usualy Java processes). It's often handy to request a stacktrace from the running process; you can do this without terminating it by issuing 'kill -3 <pid>'. Good, eh?

Unix Join with tabs

2009-04-23T09:38:00.000-07:00

This one had me scratching my head for a while yesterday. When you use the unix 'join' command with the default separators (whitespace) then you can join tab delimited files, but the output is space delimited.

I tried specifying the join character with -t "\t" or -t '\t' and even the despearate -t\t, but none of that works. Turns out you have to use an *actual* quoted tab character (which I found through a quick google search, the solution was here, thank you JJinuxLand).

You can insert a tab on the command line with the following key combo "ctrl-v <tab>".

Job's a goodun.

print "Hello World!"

2009-04-23T09:34:00.000-07:00

Hi,

I'm intending for this blog to just be a brain export of any useful/interesting or confusing things I come across in the course of my work. It'll probably contain items on MySQL, relational databases in general, Unix/Linux, Perl, Java, bash, svn, eclipse and the like. It may also end up having some interesting or surprising biology/biochemistry/bioinformatics items as well.

Paul