Bash: Piping Data into a Command Using Heredocs
Join the DZone community and get the full member experience.
Join For FreeI’ve been playing around with some data modeled in neo4j recently and one thing I wanted to do is run an ad hoc query in the neo4j-shell and grab the results and do some text manipulation on them.
For example I wrote a query which outputted the following to the screen and I wanted to sum together all the values in the 3rd column:
| ["1","2","3"] | "3" | 1234567 | | ["4","5","6"] | "6" | 8910112 |
Initially I was pasting the output into a text file and then running the following sequence of commands to work it out:
$ cat blah2.txt| cut -d"|" -f 4 | awk '{s+=$0} END {print s}' 10144679
One way to avoid having to create blah2.txt would be to echo the output into standard out like so:
$ echo "| ["1","2","3"] | "3" | 1234567 | | ["4","5","6"] | "6" | 8910112 | " | cut -d"|" -f 4 | awk '{s+=$0} END {print s}' 10144679
But it gets a bit confusing as the number of lines of results increases and you have to keep copy/pasting the cut and awk parts of the chain around which was annoying.
One of the things I read on the bus this week was a blog post going through a bunch of bash one liners and half way through it covers piping data into commands using heredocs which I’d completely forgotten about!
A simple example could be to send a simple message to cat which will output the message to standard out:
$ cat <<EOL heredoc> hello i am mark heredoc> EOL hello i am mark
That works if we want to pipe data into a single command but I didn’t know how we’d be able to pipe the output of that command to another command.
In fact it’s actually reasonably simple:
$ cat <<EOL | cut -d"|" -f 4 | awk '{s+=$0} END {print s}' pipe pipe heredoc> | ["1","2","3"] | "3" | 1234567 | pipe pipe heredoc> | ["4","5","6"] | "6" | 8910112 | pipe pipe heredoc> EOL 10144679
And now I have no need to create random text files all over my machine!
Published at DZone with permission of Mark Needham, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments