Bash: Piping Data into a Command Using Heredocs

I’ve been playing around with some data modeled in neo4j recently and one thing I wanted to do is run an ad hoc query in the neo4j-shell and grab the results and do some text manipulation on them.

For example I wrote a query which outputted the following to the screen and I wanted to sum together all the values in the 3rd column:

| ["1","2","3"]         | "3"                             | 1234567    |   
| ["4","5","6"]         | "6"                             | 8910112    |

Initially I was pasting the output into a text file and then running the following sequence of commands to work it out:

$ cat blah2.txt| cut -d"|" -f 4  | awk '{s+=$0} END {print s}'  
10144679

One way to avoid having to create blah2.txt would be to echo the output into standard out like so:

$ echo "| ["1","2","3"]         | "3"                             | 1234567    |   
| ["4","5","6"]         | "6"                             | 8910112    | " | cut -d"|" -f 4  | awk '{s+=$0} END {print s}'   
10144679

But it gets a bit confusing as the number of lines of results increases and you have to keep copy/pasting the cut and awk parts of the chain around which was annoying.

One of the things I read on the bus this week was a blog post going through a bunch of bash one liners and half way through it covers piping data into commands using heredocs which I’d completely forgotten about!

A simple example could be to send a simple message to cat which will output the message to standard out:

$ cat <<EOL
heredoc> hello i am mark
heredoc> EOL
hello i am mark

That works if we want to pipe data into a single command but I didn’t know how we’d be able to pipe the output of that command to another command.

In fact it’s actually reasonably simple:

$ cat <<EOL | cut -d"|" -f 4  | awk '{s+=$0} END {print s}' 
pipe pipe heredoc> | ["1","2","3"]         | "3"                             | 1234567    |   
pipe pipe heredoc> | ["4","5","6"]         | "6"                             | 8910112    | 
pipe pipe heredoc> EOL
10144679

And now I have no need to create random text files all over my machine!