I wrote the shell script described here ('grep4') to find up to four items at the same time in my data tables, which are big text files with one record per line. The script gives each searched-for item its own color on the terminal screen. It lets me know in advance how many 'hits' there are, so…

Big data or small data, a lot of it comes these days as plain text arranged in rows and columns. Here are a few tips on working with tables of text, both in text editors and on the command line. CSV isn't what it seems. Text tables are often exported from databases and spreadsheets as…

GNU AWK is a great tool for working with tables of data, and by using AWK arrays, you can quickly do column sums (and other calculations) based on unique values in another column (see below). Pivot tables in AWK are a bit trickier. In this article I do a slow walk through the code for a simple…

Command line wizards are forever encouraging the rest of us to learn regular expressions (regex). We're told regex is elegant, powerful and incredibly useful. The downside is that regex can be seriously fiddly. One fly-spot out of place, and the code fails. For this reason I only use 'non-simple'…

EDIT 26/04/2014 I wrote the original fandr script to do single-word replacements in big files. To do multiple-word find-and-replace, the 'needle' variables have to be double-quoted, as in the revised script below. An example of a multiple-word replacement is at the end of this article. I use two…

YAD (Yet Another Dialog) creates GUI dialogs for data entry, user notification, etc. It was written by Victor Ananjevsky and is described as a fork of Zenity. In fact, it greatly extends Zenity and has a very large number of handy options. Here I explain how to build a multiple-item data entry form…

Trove is the National Library of Australia's online database. It contains almost 400 000 000 digital items, including Australian newspaper articles from 1803 to 1954. Trove's newspaper portal is particularly useful. You can view an entire digitised newspaper page, or select (or search for) an…

If you do a Web search for 'linux delete blank lines' you'll find lots of command-line advice. Some of the advice, though, only applies to special cases. Here I explain two elegant and general methods for finding and deleting any blank lines in a text file. As a sample text I'll use the 8-line file…

Geoscience Australia has a freely downloadable gazetteer of Australian placenames with more than 370 000 entries. The download is a .zip file (83 mb) containing a user guide together with placenames data in GML, KML and Microsoft Access (.mdb) formats. What I wanted from all that data was a simple…

Here's a BASH shell script that automatically logs the date and time I start my Iceweasel web browser, and the length of time I use it. I launch the script (and Iceweasel) with a keyboard shortcut, and close the script by closing Iceweasel. Click for full size The log file has entries in the form…