delete all empty files in a directory

Automatically generating files? Annoyed at all the empty ones? Here's how to purge them:

> for file in $( ls . ); do if [ ! -s $file ]; then rm -f $file; fi; done

Of course you could just get over your fear of find and take a look at that man page. Perfect for a recursive search and delete all in one:

> find . -empty -delete

simple, no?

escaping quotes in bash variables

I often need to escape quotation characters or other special characters that are being piped into a bash process:

Say you have a mysql table which contains protein names. Some of these have the ' character in them (e.g. "Inosine 5' monophosphate dehydrogenase") and you want to do some bulk processing on these names, you could do something like this:

> mysql -u paulbo -N <<< "SELECT name from proteins" | while read protein_name; do mysql -u paulbo -N "SELECT count(*) FROM data INNER JOIN proteins ON data.protein_id = proteins.protein_id where proteins.name = '${protein_name//\'/\\\'}'"; done

Ah, three backslashes. Why didn't I think of that?

And sometimes you just want to output the text with the special characters all converted into something a bit more amenable (like the old and trusted '_').

> mysql -u paulbo -N <<< "SELECT name from proteins" | while read protein_name; do echo ${protein_name//[-\/\:\'\"\(\) ]/_};done

Excluding certain files from a directory listing

Ok, well, this will be obvious to anyone who's really read the ls man page, but I only came across it a couple of days ago.

Say you have a directory with tons of files in it mostly with a single extension and you want to see what else is in there. Sure, you can use grep, but you can also use ls's inbuilt flat --hide.

> ls -l
arg.txt
blarg.txt
foo.txt
...
interesting.sh
... # and a ton more *.txt files
zlargyblorg.txt
> ls -l --hide=*.txt
interesting.sh
hidden.csv

Isn't that useful?

batch renaming files with spaces

Why do people keep giving me tons of files with spaces in the filename?

Anyway, here's a good way to get rid of those pesky spaces:

ls * | while read file; do mv "$file" ${file// /_}; done

First I tried using "for file in `ls *`" but of course the whitespaces came back to bite me... This was also true for the mv command. You have to quote "$file" in order for the whitespace ridden filename to be recognised as a unit rather than multiple file descriptors.

bash array size

Strangely this caught me off guard. If you use ${#array[@]} to get the 'size' of an array, it actually only returns the number of assigned elements in the array.

e.g.

> array[23]=123
> echo ${#array[@]}
1

Hmm... only 1? Not 24?

As far as I can tell, there's no way around this. Just don't expect this behaviour and fill your arrays wisely.

While we're on bash arrays, remember that you can change the 'join' character for naive printing of arrays by manipulating the IFS (Internal Field Separator) variable. Below I also show that the quotation context is important for this:

> array[0]=1;array[1]=2;array[2]=3;
> echo ${array[*]}
1 2 3
> echo "${array[*]}"
1 2 3
> IFS=","
> echo ${array[*]}
1 2 3
> echo "${array[*]}"
1,2,3
> 

Note: It's best practice to store and then restore the original IFS variable.

> ORIG_IFS=IFS
... do stuff
> IFS=ORIG_IFS

Java: Heap Dump on OutOfMemoryError

You can request the jvm create a heap dump when an OutOfMemoryError is thrown. This is handy if you have a process that consumes a ton of RAM and you don't know why. Set the max heap size to something around 500M (or less. It needs to be fairly small if you're going to inspect the heap with 'jhat'). Use the -XX:+HeapDumpOnOutOfMemoryError flag to request the heap dump. This will output to java_pid.hprof by default. You can set the output filename manually using -XX:HeapDumpPath=<filename>.

e.g.

> java -Xmx100m -XX:+HeapDumpOnOutOfMemoryError -XX:HeadDumpPath=/tmp/dump.hprof com.geekbraindump.MyMemoryHoggingClass

Linux: command line cut and paste

It's always annoyed me that I have to open up a file in order to cut and paste the contents into a web browser (I use this a lot for capturing information on an internal wiki). As of 2010-03-23 there are no default command line access utilities for this (on CentOs anyway).

However, download and install xclip and the clipboard is yours to command.

xclip allows access to both the PRIMARY (middle mouse button) and SECONDARY (standard copy/paste) selections.

By default piping into xclip puts the text in the PRIMARY clipboard (middle mousebutton).

> echo $RANDOM | xclip
> xclip -o
4807

You can define which selection to input to. Say you want to store text in the SECONDARY selection (accessed using standard cut and paste commands):

> echo $RANDOM | xclip -sel 'clipboard'
> xclip -o -sel 'clipboard'
4807

You can now use edit->paste to output the text. Note that the random number was the same as before. I guess this is some shell caching mechanism. For a new random number each time you have to use a new shell:

> (echo $RANDOM) | cat
1297