Over a million developers have joined DZone.
Platinum Partner

Moving files and folders into hashed subfolders

· Performance Zone

The Performance Zone is brought to you in partnership with AppDynamics. Discover five of the top performance metrics to capture to assess the health of your enterprise Java application.

The following will move a series of files into subfolders. It hashes the file names, building a two character, two folder deep hierarchy to split the files, e.g. a1/b2. The motivation for this is to split 500,000 folders into a manageable hierarchy, to avoid file system limits – in NTFS this is quite slow, and well over the 64,000 limit in ext4.

vi f
for f in `ls pacer/`; do echo -n $f | md5sum | sed 's`\(..\)\(..\).*`\1/\2/`' | tr -d '\n'; echo $f; done;
./f > t
mkdir ../pacer2
./t | sed 's`\(..\)/\(..\)/\(.*\)`mkdir pacer2/\1\nmkdir pacer2/\1/\2\nmv pacer/\3 pacer2/\1/\2/\3`' > dirs
./dirs




The Performance Zone is brought to you in partnership with AppDynamics.  Learn the essentials of APM and how to implement best practices of application performance, better understand what it means to capture, analyze, and react to performance problems as they arise, and more specifically with .NET applications. 

Topics:

Published at DZone with permission of Gary Sieling , DZone MVB .

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}