Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Shell Scripting Notes – Searching Matching Filenames and Matching Content

DZone's Guide to

Shell Scripting Notes – Searching Matching Filenames and Matching Content

Here are a few shell scripting notes that might be useful to you.

· DevOps Zone
Free Resource

Learn more about how CareerBuilder was able to resolve customer issues 5x faster by using Scalyr, the fastest log management tool on the market. 

Some random shell scripting notes for future reference:

  • On OS X: find -E . -regex ‘pattern’

    • E specifies extended regex support
  • On Linux flavors: find . -regextype posix-regex -regex ‘pattern’

  • Posix vs basic vs extended regex character class differences.

  • Pipe result to newfile > : eg grep ‘pattern’ file > output.txt

  • Pipe result appending to file > : eg grep ‘pattern’ file >> output.txt

  • Capture output as String? : $(some expression)

Iterate files:

for f in some-file-pattern or something producing a list of files
do ... done

Use find . -name ‘pattern’ to recurse matching files down subdirs

Find with a regex for multiple patterns:

find -E . -regex ".*ext1|.*ext2|.*ext3"

first line of file:

head -n 1 filename

grep -o : only display match

Match patterns in file and output matches or matched groups:

#!/bin/bash
for f in `find -E . -regex ".*ext1|.*ext2|.*ext3"`
do
line=$(head -n 1 $f)
if [[ $line =~ TextAtStartOfLine:[[:space:]]*([[:digit:]]+) ]]; then
echo "${BASH_REMATCH[0]}" # print whole match
echo "${BASH_REMATCH[1]}" # print 1st group match
fi
done

View rawgistfile1.txt hosted by GitHub

Match files, patterns in files, and pipe matches to file:

#!/bin/bash
for f in $(find -E . -regex ".*ext1|.*ext2|.*ext3")
do
echo $f $'\n' $(egrep -o 'TextAtStartOfLine:[[:space:]]*[[:digit:]]+' $f) $'\n' >> report"$(date)".txt
done

View rawgistfile1.txt hosted by GitHub

Find out more about how Scalyr built a proprietary database that does not use text indexing for their log management tool, allowing customers to search 1TB of data in under a second. 

Topics:
linux ,script ,shell scripting

Published at DZone with permission of Kevin Hooke, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}