When writing or editing scientific-type text in LibreOffice Writer, I rely a lot on its spellchecker. Unfortunately, the scientific terms I'd like to check aren't in the default dictionaries behind LibreOffice's spellchecking routine. There are two ways out of this dilemma. The first is to deal…

In my data-cleaning work I often make up tallies of selected individual characters from big, UTF-8-encoded data files. What's the best way to do this? As shown below, I've tried grep/sort/uniq, AWK and Ruby, and AWK's the fastest. The trials also revealed an unexpected problem with the uniq program…

My wife has been in the same Scottish Country Dancing group for many years, and back in 2005 she began keeping a record of the dances done at each meeting. She lists a fair bit of information in her "dances done" spreadsheet (screenshot below). She recently asked me "Would it be…

I carry a Garmin Etrex 10 when I go bug-collecting. It's a handheld GPS receiver without many frills, and I only use it to record the latitude and longitude of each of my collecting sites as a waypoint. At the start of a field trip I delete all previous waypoints. When I do this, the trip's…

I've written several BASH scripts that use GNU AWK's printf function to round off decimal numbers. Now I've learned that the rounding is biased, and I need to re-do the scripts. For me the desirable way to round off a decimal is to round up if the bit to be discarded is greater than one half, and…

SANITY WARNING: Please don't read this article unless you're a compulsive shell scripter, like me. Seriously. There are some excellent FOSS sequence viewers in the distro repositories, with great GUIs and lots of features. Why would you want to build a less capable viewer that only works in a…

When cleaning UTF-8 text files I sometimes come across invisible characters that I call 'gremlins'. These aren't the usual non-printing characters, like whitespace and (horizontal) tab, which are non-printing characters I expect to find in the plain text files I work with. Gremlins are weird things…

Recently I was checking for unmatched braces in a big text file with nearly half a million lines (300+ MB). Was there a closing brace for every opening brace? The text file was called 'beetle' and was actually a tab-separated table with each line divided into 77 fields. What I wanted to do was find…

Flags in AWK are variables which are set to either true or false. They're handy for defining ranges over which AWK can act, as shown below. The AWK used here is GNU AWK 4.1.1 (gawk 4). Sometimes flags aren't needed I'll demonstrate with a simple text file called demo, which has 6 lines with 3…

Lexical illusions are very hard to find when proofreading. The most common lexical illusion is a duplicated word, as in this well-known example: A lexical illusion: many people are not aware that the the brain will automatically ignore a second instance of the word 'the' when it starts a new line. …