Unix lessons: sed

sed is the binary file for the Stream EDitor program present in almost all Unix-like systems distributions. Its purpose it to perform selection, editing and removal commands over a stream of text; given the stream nature of almost anything in Unix, sed can manipulate files and the output of other commands, which make it one of the incarnations of the Unix philosophy : 

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

sed can be piped the output from any command to filter it, or to say it in functional terms to map a command to it.

Commands

By default, commands are applied in a line-wise fashion. For example, the a command adds a line of text after each matched line.

$ ls -1 /bin/ | sed -e 'aAFTER' | tail -n 4
zmore
AFTER
znew
AFTER

The d command deletes the line that match a particular pattern; in this case, the lines that start with a z.

$ ls -1 /bin/ | sed -e '/^z/d' | tail -n 4
vdir
vmmouse_detect
which
ypdomainname

A more flexible line-wise replacement is the famous s:

$ ls -1 /bin/ | sed -e 's/^z//' | tail -n 4
grep
less
more
new

which, as you can see, matches regular expressions, not just fixed strings.

Filtering

Also by default, sed does reprints on the standard output every line that is not matched by the commands. This mode is useful when you just need to edit some lines and leave unaltered the rest; however, silent mode is a better choice for performing selections:

$ ls -1 /bin/ | sed -n -e 'p' | tail -n 4
zgrep
zless
zmore
znew

This time, we have told sed to operate in silent mode with -n. However, the command passed to it by -e was p, which means just to print every line. If we were to remove the silent mode switch, this output would be printed:

$ ls -1 /bin/ | sed -e 'p' | tail -n 4
zmore
zmore
znew
znew

which seems redundant. sed is printing each line in the input and reprint it after that due to the p command.

Now that we are able to not print anything by default, we can add selectors to p to limit what we are considering in the input. For example, we can select just the lines matching an expression in a grep fashion:

$ ls -1 /bin/ | sed -n -e '/sh$/p'
bash
dash
rbash
sh
static-sh

Or we can print from the 5th to the 8th line:

$ ls -1 /bin/ | sed -n -e '5,8p'
bzcmp
bzdiff
bzegrep
bzexe

Line numbers start from 1 as in all Unix editors. The special selectors for the last line is $.
We can also select the lines by pattern instead of number, which is very handy to parse multiple-line logs:

$ cat /var/log/onebip/onebip/log-*mailman-sms-connectivity* | sed -n -e '/REQUEST BODY/,/RESPONSE CODE/p' | tail -n 4
2013-10-11T15:21:01  127.0.0.1 REQUEST BODY  {"country":"ES","description":"Super powers","container":"purchase\/999235"}
2013-10-11T15:21:01  127.0.0.1 RESPONSE CODE HTTP/1.1 201 Created
2013-10-11T15:21:03  127.0.0.1 REQUEST BODY  {"country":"IT","description":"Super powers","container":"purchase\/1188829327"}
2013-10-11T15:21:03  127.0.0.1 RESPONSE CODE HTTP/1.1 201 Created

since the selection can be performed multiple times, matching each pair of REQUEST BODY and RESPONSE CODE in the file (be aware this may not scale to higher throughputs if logs can overlap each other.)

Putting it all together:

$ cat /var/log/onebip/onebip/log-*mailman-sms-connectivity* | sed -n -e '/REQUEST BODY/,/RESPONSE CODE/p' | sed -e 's/.*127.0.0.1//g' | tail -n 2
 REQUEST BODY  {"country":"IT","description":"Super powers","container":"purchase\/1188829327","price":100,"smsc":"Mpayit"}
 RESPONSE CODE  HTTP/1.1 201 Created

Conclusions

sed is in the basic toolbox of the Unix way; with it you can edit a stream of test by substituting expressions and filtering from one to multiple lines. It can be used in concert with grep where the latter does the heavy job of selecting while sed edits line-wise.

Moreover, its interface is consistent with the commands of Vim (s, p, d commands), and it should not be difficult to learn with a bit of practice. What I find helpful when reading tutorials of this kind is to type in every command I want to learn in my own terminal, as practicing leads to better retention than reading.