Lexicographically Sorting Large Files in Linux

when i hear the word “sort” my first thought is usually “hadoop”! yes, sorting is one thing that hadoop does well, but if you’re working with large files in linux the built-in sort command is often all you need.

let’s say you have a large file on a host with 2gb or more of main memory free. the following sort command is a efficient way to lexicographically -order large files.

lc_collate=c sort --buffer-size=1g --temporary-directory=./tmp --unique bigfile.txt

let’s break this command down and examine each part in detail.