tag:blogger.com,1999:blog-28814972498516449092024-03-05T06:22:42.888-08:00GeekBrainDumpThis is mainly somewhere for me to dump interesting tidbits of tech info. Stuff that I may forget but will need again in the future. For a proper blog, visit my electronics blog (http://www.fangletronics.com) or my wife's pre-school crafty blog (http://www.filthwizardry.com).PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.comBlogger96125tag:blogger.com,1999:blog-2881497249851644909.post-78915374213709201932022-11-02T09:57:00.000-07:002022-11-02T09:57:35.131-07:00CITYSPORTS E06 Error - fixI have one of these treadmills. A few times it has stopped with the belt unable to move but with the display still working. The last time it displayed an error message: "E06" which means that the motor is overloaded.
The first time this happened I thought I was going to have to replace the treadmill. I took it apart for an inspection and noted that the motor has an access port for the brushes. I cleaned the brush and put it back and the treadmill worked again for about 1 month before the same error was thrown. I ordered some replacement brushes and replaced the worn down one - now everything is working just fine.<div class="separator" style="clear: both;"><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg9g1xokNWopoD7HdT6iQbh-8hVitZr6eKrxgJ09m-x7ZDyPgsxLMMG1ZtHmVUL4o0KkpaMHR2FGdA-6XwvB9DA6qPWl6JEMeMy2qZyfDjG4jHu7tuJ0jMfmBm3fnFta-Y4jd1pGdoBCAYuYeAT1NP7WxGpkECZxwXLwI3p8qybWU8dn5h3265dYeLYA/s4080/PXL_20221014_090808050.jpg" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="3072" data-original-width="4080" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhg9g1xokNWopoD7HdT6iQbh-8hVitZr6eKrxgJ09m-x7ZDyPgsxLMMG1ZtHmVUL4o0KkpaMHR2FGdA-6XwvB9DA6qPWl6JEMeMy2qZyfDjG4jHu7tuJ0jMfmBm3fnFta-Y4jd1pGdoBCAYuYeAT1NP7WxGpkECZxwXLwI3p8qybWU8dn5h3265dYeLYA/s320/PXL_20221014_090808050.jpg" width="320" /></a></div><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUymr28xB1XM1-oZ8oD9em7O3T9_B90zVQPMpANx9ez10nNIvzf_-fiEQfUQ412fERNIn5IYeiRuuTRH_m0N-vGpbPQhJARObihROUU_PbAPIfij9K-QCna08FDpM7ouBPDls6WKvydy_OzhvATlVpfXmNhrzZP3OoSyjDAbvK4y14GRy-zyfnt7RKSA/s4080/PXL_20221014_090900028.jpg" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="4080" data-original-width="3072" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUymr28xB1XM1-oZ8oD9em7O3T9_B90zVQPMpANx9ez10nNIvzf_-fiEQfUQ412fERNIn5IYeiRuuTRH_m0N-vGpbPQhJARObihROUU_PbAPIfij9K-QCna08FDpM7ouBPDls6WKvydy_OzhvATlVpfXmNhrzZP3OoSyjDAbvK4y14GRy-zyfnt7RKSA/s320/PXL_20221014_090900028.jpg" /></a><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeQtE7V5jEvnDPlCGvVICddOWuzBMv3-SOsXmw4yzssDt2lAOkHp2CYEM0sGQ230at2eZc-FHojpbmbOcmMUzdJLQZfem7zmRzW0DQSwOt0WM_myDZQTHs-IHh372B--gfNSIVd4JcdVjWBEPu2QjPiFJaqv1R8BAjCh0ZNHDd307vtPE9pBpA_-zDXA/s4080/PXL_20221014_091216862.MOTION-02.ORIGINAL.jpg" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" data-original-height="3072" data-original-width="4080" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeQtE7V5jEvnDPlCGvVICddOWuzBMv3-SOsXmw4yzssDt2lAOkHp2CYEM0sGQ230at2eZc-FHojpbmbOcmMUzdJLQZfem7zmRzW0DQSwOt0WM_myDZQTHs-IHh372B--gfNSIVd4JcdVjWBEPu2QjPiFJaqv1R8BAjCh0ZNHDd307vtPE9pBpA_-zDXA/s320/PXL_20221014_091216862.MOTION-02.ORIGINAL.jpg" width="320" /></a></div></div>
In this last picture you can see how worn down the old brush was!
<div style="text-align: center;"><img alt="" border="0" data-original-height="3072" data-original-width="4080" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipt_bAtoeqxMPaNoFBAC8XqBS9ZDklM-nVm0Tm_m-8acb0d2U3DbdHwGGLKGb7V-idNSTf6A_rbEWIFWQHTmFz0hXstXUmdWbTPEhKb36lYRNNDORpSxV8r4m99SFN5_fKVFfXg0sWZVDtix7b-8peVOByBI2LhF61aPjU-MHO6a5tMvIRK7uB2iQNTA/s320/PXL_20221022_102906187.jpg" width="320" /></div></div>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-67603663976605842752019-03-07T10:42:00.000-08:002019-03-07T13:18:18.543-08:00R: use purrr::map_dfc to update all cells in a data.frameI've just loaded a large data.frame generated from a set of SDFs from a collaborator. The 'empty' cell value is set as a period ('.') which isn't great for general analysis.<br />
<br />
The quick way of converting these to <code><b><i>NA</i></b></code> is by using the <code><b><i>map_dfc</i></b></code> function from the <code><b><i>purrr</i></b></code> package.<br />
<br />
<br />
<script src="https://gist.github.com/paulboardman/9abca3bd0ff20348048379f87f4123b2.js"></script>
This should output the following (note that all occurrences of '.' have been replaced with <code>NA</code>):<br />
<pre>
a b c
1 T 2 morty
2 D 1 summer
3 P 1 rick
4 O . summer
5 V 3 summer
6 M 3 summer
7 Z . rick
8 F 2 .
9 J 2 rick
10 E 3 rick
a b c
1 T 2 morty
2 D 1 summer
3 P 1 rick
4 O NA summer
5 V 3 summer
6 M 3 summer
7 Z NA rick
8 F 2 NA
9 J 2 rick
10 E 3 rick
</pre>
PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-47242584925935825322018-01-30T09:22:00.000-08:002018-01-30T09:22:10.691-08:00Count SDF entries in a fileThis is something I find myself doing a lot, so thought I'd share. There's a quick and easy way to count the number of entries in an SDF when you're on the command line:
<script src="https://gist.github.com/paulboardman/489e603db8dacd6d5bced782324a8a91.js"></script>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-11176348571787452552017-09-05T02:10:00.004-07:002017-09-05T02:11:04.652-07:00capture logging info from xargs comandHere's an example of how to capture STDOUT and STERR from a set of commands parallelized using '<span style="font-family: "courier new" , "courier" , monospace;">xargs</span>':<br />
<br />
<br />
<script src="https://gist.github.com/paulboardman/615b72f761762d6119b02dd678d67a7d.js"></script>
Note that I'm using '<span style="font-family: Courier New, Courier, monospace;">find</span>' rather than the output of '<span style="font-family: Courier New, Courier, monospace;">ls</span>' as the input (since '<span style="font-family: Courier New, Courier, monospace;">find</span>' will not return anything if no files are found). You can define multiple shell arguments in this way and refer to them in the normal manner ("<span style="font-family: Courier New, Courier, monospace;">$1</span>", "<span style="font-family: Courier New, Courier, monospace;">$2</span>", "<span style="font-family: Courier New, Courier, monospace;">$3</span>", etc.).PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-53664692756105320852017-02-14T06:24:00.001-08:002017-02-14T06:24:13.604-08:00R: find the color name you wantSimple tip for finding the color name you want from the base set.<br />
<br />
<script src="https://gist.github.com/paulboardman/3d8c267eb2f6306f5f876dbdeb679b4a.js"></script><br />
<br />
This gives the following output:<br />
<br />
[1] "darkorange" "darkorange1" "darkorange2" "darkorange3" "darkorange4" "orange" "orange1" <br />
[8] "orange2" "orange3" "orange4" "orangered" "orangered1" "orangered2" "orangered3" <br />
[15] "orangered4"
PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-30335407728949489612017-01-23T09:00:00.000-08:002017-01-23T09:00:46.269-08:00Terminate a frozen SSH sessionI work remotely via a VPN and am often plagued by frozen SSH sessions (especially if I step away from my desk for any length of time). I always used to close the shell/terminal - which is a bit annoying. Then I came across the solution below, just type enter (in the frozen shell) then tilde and period/full-stop and you drop out of the ssh session.<br /><br />
<script src="https://gist.github.com/paulboardman/c9e6019e7b97bc4accfa172665a2cf6d.js"></script>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-29368845548614179372017-01-09T06:21:00.000-08:002017-01-09T06:21:31.721-08:00Generating random numbers in bashI wanted to assign each member of a training set to a cross-validation fold. I wanted to do this in a bash script and so ended up using something like the following:<br />
<script src="https://gist.github.com/paulboardman/e32addbe1715c027e8b231e28cd088ac.js"></script><br />
This script generates <em>n</em> numbers selected randomly in the series of integers from 1 to <em>m</em>. In this case <em>m</em> represents the number of cross validation folds.<br />
<br /><br />
You could generate a random CV fold assignment like so:<br />
<script src="https://gist.github.com/paulboardman/1cb679893363ff39be41233e54731737.js"></script><br />
Depending on your input file format of course.PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-3628250776260669722016-08-05T03:15:00.000-07:002016-08-05T03:15:18.101-07:00Minecraft Stop-Motion Movie Creator - too many frames<p>This post is a work in progress.</p>
<p>My kids love using the "Minecraft Stop-Motion Movie Creator". They've just finished putting together an epic animation (> 400 frames) and went to save the movie but were greeted with "Too many frames - please remove 91 frames". They were crestfallen. Surely this limit should be advertised somewhere?</p>
<p> Anyway, I attached the device to my computer to take a look - with the idea of downloading all the frames myself and using different software to stitch them together.</p>
<p>
This was done on a Mac - I used <a href="https://www.android.com/filetransfer/">androidfiletransfer</a> to connect and browse the device (a Hudl2 in this case). <br/>
For any of you who want to do the same the relevant files are in: <em>Android->data->com.mattel.MCSMMC->files->Movie</em>. Each movie has it's own number and each frame of each movie is in an individual, numbered, folder. e.g. Movie->9->Frame->11 contains the files for frame 11 of movie 9.</p>
<p>I'll update this post when I've figured out how to resolve the issue - probably by creating the movie using separate software or by splitting the movie on the device.</p>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-38740517881643393162016-07-26T03:11:00.000-07:002016-07-26T03:11:47.914-07:00List user installed R packagesIt's handy having a list of packages you've installed on hand.
<p> This script will output the package name and version number:
<script src="https://gist.github.com/paulboardman/30865bcb14aa87adcdb8b79f3d57c28f.js"></script>
</p>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-12987383677803028802016-01-17T23:08:00.003-08:002016-03-28T02:51:12.497-07:00Evernote sync error - "the content is invalid"I've been getting these on and off for months. It's been a pain to try and sort out what's going on and there seems to be nothing in the Evernote forums that's particular to my issue. I have a set of notes that just won't sync with nothing obviously wrong with them. The error in the log looks like this:
<blockquote>
-[ENSyncEngine(Errors) addSyncErrorForObjectWithID:title:message:underlyingError:] | Adding error: Error Domain=ENSyncEngineErrorDomain Code=-2000 "The content is invalid." UserInfo=0x... {underlyingError=<EDAMUserException: 0x...; errorCode = 11; parameter = The content of element type "en-todo" must match "EMPTY".;>, NSLocalizedFailureReason=Could not sync note ‘Work Done: 2015-12-09 (Wed)’., localObjectID=0x... <x-coredata://2CA93062-377E-4FB9-BAC5-.../ENNote/p3120>, NSLocalizedDescription=The content is invalid.}
</blockquote>
<p>So, this makes it sound like there's an issue with a 'TODO' element (checkbox?) and that the tag, for some reason, contains content. Looking at the XML export of that particular note doesn't show anything amiss/unusual around the TODO/checkbox elements.
</p>
<p>
<b>Solution</b>: In my case, the issue turns out to be the emoticons - I use smileys and Evernote helpfully converts the ASCII versions into lovely little images. These seem to cause the invalid content issue. If I highlight the areas around the emoticons and 'simplify formatting' (Formatting dropdown) the note will now sync just fine.
</p>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-25009425344695611332015-11-17T02:40:00.000-08:002015-11-17T02:40:00.210-08:00Ergotron Workfit-S - fix for reduction in range of motion<p>I've been using one of these Ergotron standing desks for a few years now: <a rel="nofollow" href="http://www.amazon.com/gp/product/B003Y3PF6E/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=B003Y3PF6E&linkCode=as2&tag=fangletronics-20&linkId=6VETIA6QK4LNVJLX">Ergotron WorkFit-S 33-341-200 </a>.<img src="http://ir-na.amazon-adsystem.com/e/ir?t=fangletronics-20&l=as2&o=1&a=B003Y3PF6E" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> The maximum height it'd raise to has reduced over time; it took me a while to realize it as this was quite a slow process. </p>
<p>
It turns out the fix is really simple. I found this entry in the Ergotron forums: <a href="http://forum.ergotron.com/viewtopic.php?f=13&t=2904">WorkFit-S Dual Maximum height</a>.</p><p></>The main point is in the second comment (a reply from an Ergotron tech):
<blockquote>
One thing that you can try is "resetting" the WorkFit-S. To do this, lower it with one hand on the handle & one hand on the workstation surface - when it gets to the low point, push on the workstation surface a little bit - you should hear the sound of the desk "reset". Afterwards, you should be able to have the full range of motion that you had before.
</blockquote>
</p>
<p>
This worked well for me - at least after I realized the 'workstation surface' was really anywhere on the movable arm (my Ergotron doesn't have a workstation surface). I had to press quite hard, the arm moved further down than it's base resting position and then had the full range of motion back. This meant that standing was comfortable again (before, my arms were bent as the keyboard tray was too low).
</p>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com7tag:blogger.com,1999:blog-2881497249851644909.post-87744741311155989672015-11-12T03:21:00.000-08:002015-11-12T03:21:42.769-08:00Git: remove files from the index which are ignored by .gitignore<p>If you update your patterns in <em>.gitignore</em>, you're often left with a set of files in the index and repository that should be removed. This can be achieved as follows:</p>
<p>
<script src="https://gist.github.com/paulboardman/34ca445b95026d3450b3.js"></script>
</p>
<p>You now have the ignored files staged for deletion.</p>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-70530159302190751442015-02-25T05:14:00.000-08:002015-02-25T05:14:22.046-08:00RMarkdown - making and using a general Rmd file for re-use<p>There are some tasks I do a lot and it'd be good to have general .Rmd documents available to report on the outcome of those tasks. Some examples would be a standard report on an ML experiment, or an overview from a data curation pipeline. I've been using rmarkdown for generating reports, but it has some limitations for this. I don't want to have a new Rmd file for every report. I'd like to have a configurable template file that lives in a shared location that I can run each time.
</p><p>
Unfortunately, if you try to do this you'll soon find that rmarkdown expects the Rmd file to be in your working directory - well, the working directory is set to the location of the Rmd file. See the following line in <code>render.R</code>: <a href=https://github.com/rstudio/rmarkdown/blob/4c8b860f276bcceb9fe10722544ee189d1ae7548/R/render.R#L96>github-render.R_L96</a></p>
<p>
My current work around for this (and please let me know if you have a better way) is to have a helper script which copies over the Rmd file to the current working directory, generates the report and then removes the temporary files:
<script src="https://gist.github.com/paulboardman/7ea46c99e5929f0e5ac4.js"></script>
The Rmd file has a line for loading in the configuration:<br/>
<code>
library(yaml)<br/>
config <- yaml.load_file("model_report.yaml")</code>
</p>
<p>
The <a href=http://yaml.org>YAML</a> configuration file contains the project specific variables (e.g. output suffix, flags for adding/subtracting items from a report, important dir locations). This file could easily be generated on the fly via arguments to the bash script.
PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-82102206789816923622015-02-18T01:56:00.001-08:002015-02-18T01:56:26.366-08:00Rmarkdown: automatically select correct output type for xtable I was initially setting the xtable output type manually - either globally or in each print statement. This got annoying very quickly when switching between HTML and PDF outputs. This is how you get xtable to automatically use the correct output format:
<script src="https://gist.github.com/paulboardman/93af9da20be5d9bcbb39.js"></script>
PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-6653429420083073072015-02-04T15:04:00.000-08:002015-02-06T04:29:19.347-08:00Fix Excel Office for Mac 2011 TSV output<script src="https://gist.github.com/paulboardman/12d6f071aa60b8e28230.js"></script>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-21078143568596071652015-02-04T06:25:00.000-08:002015-02-04T06:29:42.703-08:00RMarkdown: Setting global chunk optionsThe easiest way to do this is to have a chunk at the start of your document that isn't included in the output of the report:
<script src="https://gist.github.com/paulboardman/bb07302ca31a6720b558.js"></script>
In this example I'm setting the default behaviour so that the code isn't printed (<code>echo=FALSE</code>) and no warning messages are reported in the output document (<code>message=FALSE</code>). I'm also setting some xtable options (for easily printing out nice tables in the report).
PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com1tag:blogger.com,1999:blog-2881497249851644909.post-55314143965684166302014-07-15T03:27:00.000-07:002014-07-15T03:27:27.811-07:00NX screen sizing issueI recently got a MacBook Pro (yes yes, look at me!) - I've been going to and from the office with it whilst running NX. Turns out NX doesn't like this very much because I'm connecting and disconnecting monitors so the available real-estate keeps changing. The last time NX just stopped resizing correctly and I was left with a view of ~1/8th of my remote screen... Turns out you can access the NX display settings using "Alt-Ctrl-0" - make sure the "remote resize" option is enabled. This fixed the issue for me.PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-33858757911708493602014-04-10T06:28:00.001-07:002014-04-10T06:28:39.438-07:00Freeing space from deleted files<p>I just deleted a 33 GiB error log that had filled up my NFS shared disk. This didn't immediately free up any space on the device. This is because the file was being held open by another process and so the space can't be freed until those processes terminate and close their connections to the file. <br/><br/>To check for this condition you can use <code>lsof</code> and then <code>grep</code> for deleted files. I found what was keeping my file open and killed that process (it was a "grep -c" I'd forgotten I'd set off ages ago).
</p>
<script src="https://gist.github.com/paulboardman/10381576.js"></script>
<p>
The process keeping the file open is listed in the first column.<br/>
<br/>
For example, in the output below the grep process is holding my stackTraces.out file open:<br/>
<B><code>"grep 1234 paul 3r REG 202,166 35110973440 3285868 /nfs_share/proj/stackTraces.out (deleted)"</code></B>
</p>
PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-23733539250389592332013-12-16T06:39:00.000-08:002013-12-16T06:40:07.465-08:00KNIME: duplicate column issue Sometimes if you try to read in multiple SDFs with the "SDF Reader" node and "extract all properties" at the same time (in order to extract all the associated tag data) the node can fail with:
<p>
"Execute failed: Duplicate column name "<colname>" at positions 1 and 2" - (the column names and reported positions depend on the format of the input data).
</p>
I tried to fix up the columns thinking this was a whitespace issue or some such but had no joy. For me the solution was to <b>read the molecules without extracting the data and then use the "SDF Extractor" node to grab the data</b>. This worked with no issues.PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-87188079179510210652013-11-19T09:14:00.000-08:002013-11-19T09:14:06.168-08:00ShrewSoft VPN Manager window does not show under 64Bit Windows 7<p>I'm using version 2.2.2 of the <a href="https://www.shrew.net/">ShrewSoft VPN client</a> on my 64bit Win7 laptop. For some reason, the 'Manage' window doesn't show in this version - which bit me when my VPN details changed. The .pcf configuration files aren't re-read if you edit them and so you have to have access to the 'manage' dialog in order to update your connection - you can't even add a connection without accessing this dialog.
</p>
<p>
So, the solution for me was actually quite simple -
<ol>
<li>try to bring up the 'Manager' dialog (right click the ShreSoft icon in the bottom right of the taskbar and click on 'Manage')
<li>bring up the task manager (<code>Ctrl-Shift-Esc</code> is a handy short-cut for that) - the VPN manager should be present in the "Applications" window
<li>right click this and select 'maximize'
</ol>
It might not be pretty, but at least it's now accessible.
</p>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com7tag:blogger.com,1999:blog-2881497249851644909.post-3967749745492589512013-10-24T06:26:00.000-07:002013-10-24T06:28:13.618-07:00Compare a local and a remote file using 'diff' and process substitution.I don't think I've posted this one before:
<script src="https://gist.github.com/paulboardman/7137209.js"></script>
I've been doing a lot of moving files around between local and remote servers and have made heavy use of the above idiom. First find the files that need to be manually inspected (<code>rsync -ilvrn <localdir> <remotedir> </code>) and then use the above diff command to compare them.PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com1tag:blogger.com,1999:blog-2881497249851644909.post-85365296028416165312013-06-21T03:28:00.001-07:002013-06-28T03:04:44.757-07:00Better tab use with bashI use tab delimited files a lot. I hate comma separated values (and if you've ever had to deal with fields that could contain commas, you'll agree). If you like to use a good set of linux command line tools, then tabs sometimes cause problems.
<br/><br/>
One of the most viewed posts on this blog is the <a href="http://geekbraindump.blogspot.co.uk/2009/04/unix-join-with-tabs.html">unix-join-with-tabs</a> one where I describe the "Ctrl-v <tab>" method of inserting a tab character on the command line for things like <code>cut -f':' --output-delimiter=<tab></code> where the output delimiter needs to be a tab.
<br/><br/>
However, there's a much nicer/easier way of specifying a tab character in bash: <code>$'\t'</code>. Take a look a the QUOTING section of the bash manpage for all the details.
<br/><br/>
Here's a simple example of it in action:
<br/><br/>
<script src="https://gist.github.com/paulboardman/5830333.js"></script>PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-80175503389873170822013-06-04T09:08:00.000-07:002013-10-10T09:00:32.273-07:00Using 'find' to list files with multiple suffixesI'm going through another data cleanup session in a (very) old work directory tree. I found myself examining/compressing/removing files with a recurring set of suffixes. The '<code>find</code>' command can make this all a lot less painful.
<br/>
<br/>
Here's the command for matching a single suffix:
<br/>
<script src="https://gist.github.com/paulboardman/5707151.js"></script>
<br/>
But editing this command-line to match the next suffix of interest becomes tedious very quickly. Thankfully, you can chain together file tests like so (note the grouping):
<br/>
<script src="https://gist.github.com/paulboardman/5707161.js"></script>
<br/>
Then acting on these files is easy - just update the -exec action to what you want (e.g. "<code>-exec bzip2 {} \;</code>" - you probably want to use <code>xargs</code> or "<code>-exec bzip2 {} +</code>" for this to reduce the number of command invocations)
<br/>
<br/>
An interesting note here is that the following command isn't executed as you might expect. The '<code>ls</code>' command is only executed on the *.tsv files due to the way the expression is evaluated: from left to right with the implicit '<code>-and</code>' between the second '<code>-name</code>' and '<code>-exec</code>' exec having higher precedence than the '<code>-or</code>' between the two '<code>-name</code>' tests..
<br/>
<script src="https://gist.github.com/paulboardman/5713687.js"></script>
PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-26106741474198341872013-04-10T06:55:00.002-07:002013-10-10T09:00:57.566-07:00Finding (and fixing) files with undesirable permissionsI use a few programs that create files that are not group read/writable; this is an issue when working in a group environment (especially with Knime where important files are locked this way which prevents other users from even opening your workspaces).
Here's a find command that'll locate these files:
<script src="https://gist.github.com/paulboardman/5354789.js"></script>
This command executes 'ls -lh' on each of the files to show the current permissions (would probably be more efficient to push into xargs to run ls on multiple files at once).
The next find command will update the permissions on those files:
<script src="https://gist.github.com/paulboardman/5354835.js"></script>
Of course, you can just run chmod recursively (<code>chmod -R g+rw *</code>), but this isn't always what you want.PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0tag:blogger.com,1999:blog-2881497249851644909.post-17915790712872114722013-03-08T08:59:00.000-08:002013-03-08T08:59:16.397-08:00Merge images I have a couple of sets of related density plots (each set with ~9 images) that I need to enter into a report. I'm feeling lazy today and don't want to go through the pain of uploading each of these to our internal wiki so I merged them instead.
<p>
The <a href="http://www.imagemagick.org/Usage/montage/">montage </a>program (part of the ImageMagick suite of tools) is really simple and effective for this:
</p>
<script src="https://gist.github.com/paulboardman/5117946.js"></script>
Hey presto a set of merged density plots laid out into two columns (regardless of the number of images) with some spacing between them and a bit of drop shadow to visually separate the plots.PaulBohttp://www.blogger.com/profile/14617260943732454939noreply@blogger.com0