Lexicographically Sorting Large Files in Linux
by
Join the DZone community and get the full member experience.
Join For Freewhen i hear the word “sort” my first thought is usually “hadoop”! yes, sorting is one thing that hadoop does well, but if you’re working with large files in linux the built-in sort command is often all you need.
let’s say you have a large file on a host with 2gb or more of main memory free. the following sort command is a efficient way to lexicographically -order large files.
lc_collate=c sort --buffer-size=1g --temporary-directory=./tmp --unique bigfile.txt
let’s break this command down and examine each part in detail.
Published at DZone with permission of Alex Holmes, DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.
Trending
-
How To Check IP Addresses for Known Threats and Tor Exit Node Servers in Java
-
File Upload Security and Malware Protection
-
Merge GraphQL Schemas Using Apollo Server and Koa
-
RBAC With API Gateway and Open Policy Agent (OPA)
Comments