Lexicographically Sorting Large Files in Linux
Join the DZone community and get the full member experience.Join For Free
when i hear the word “sort” my first thought is usually “hadoop”! yes, sorting is one thing that hadoop does well, but if you’re working with large files in linux the built-in sort command is often all you need.
lc_collate=c sort --buffer-size=1g --temporary-directory=./tmp --unique bigfile.txt
let’s break this command down and examine each part in detail.
Published at DZone with permission of Alex Holmes, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.