DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. Store One Billion Files in Alluxio 2.0

Store One Billion Files in Alluxio 2.0

We look at how this open source framework can handle larger namespace volumes as people mount more storages into it and make greater use of object storage.

Andrew Audibert user avatar by
Andrew Audibert
·
Apr. 11, 19 · Analysis
Like (3)
Save
Tweet
Share
7.62K Views

Join the DZone community and get the full member experience.

Join For Free

Alluxio is a virtual distributed file system that enables applications to access files and objects in different external storage like S3 or HDFS in a unified file system namespace with a single API. Scaling the capacity of Alluxio metadata service is vital to Alluxio for a couple of reasons:

  1. Alluxio provides a single namespace where multiple storage systems can be mounted. So the size of Alluxio's namespace needs to match the sum of the sizes of all mounted storages.
  2. Object storage is increasing in popularity, and object stores often hold many more small files compared with file systems like HDFS.

In Alluxio 1.x, the metadata service is limited to around 200 million files in practice. Scaling further would cause garbage collection issues due to the limited JVM heap size of the Alluxio master process. Also, storing 200 million files would require a large memory footprint (around 200GB) of JVM heap in a single machine running Alluxio master.

The metadata service in Alluxio 2.0 is designed to support at least 1 billion files with a significantly reduced memory requirement.  To achieve this, we added support for storing part of the namespace off-heap by RocksDB on disk. Recently-accessed file system metadata is stored in memory, while older data ends up on disk. This tiered metadata storage significantly reduces the memory requirements for serving the Alluxio namespace and also takes pressure off of the Java garbage collector by reducing the number of objects it needs to deal with.

1 Setting Up Off-Heap Metadata

Off-heap metadata is the default in Alluxio 2.0, so no action is required to start using it. If you want to go back to storing all metadata in-memory, set

alluxio-site.properties

alluxio.master.metastore=HEAP
# default is ROCKS for off-heap

The default location for off-heap metadata is ${alluxio.work.dir}/metastore, which is usually under the alluxio home directory. You can configure the location by setting

alluxio-site.properties

alluxio.master.metastore.dir=/path/to/metastore

In-Memory Metadata Storage

While Alluxio has the ability to store all of its metadata on disk, this would result in reduced performance due to the relative slowness of disk compared to memory. To alleviate this, Alluxio keeps a large number of files in memory for fast access. When the number of files grows close to the maximum cache size, Alluxio evicts the less-recently-used files onto disk. Users can trade off between memory usage and speed by setting

alluxio-site.properties

alluxio.master.metastore.inode.cache.max.size=10000000

A cache size of 10 million requires about 10GB of heap.

2 Sizing the Resource

Memory

For the best performance, the metadata of the working set should fit within the master's memory. Estimate the number of files in your working set, and multiply by 1KB per file. For example, for 5 million files, estimate 5GB of master memory. To configure master memory, add a JVM option in conf/alluxio-env.sh

alluxio-env.sh

ALLUXIO_MASTER_JAVA_OPTS+=" -Xmx5G"

If you have the memory available, we recommend giving the master a 31GB heap in production deployments.

alluxio-env.sh

ALLUXIO_MASTER_JAVA_OPTS+=" -Xmx31G"

Update the cache size to match the amount of memory allocated:

alluxio-site.properties

# 31 million
alluxio.master.metastore.inode.cache.max.size=31000000

This will accommodate most workloads, and avoids the need to switch to 64-bit pointers once the heap reaches 32GB.

Disk

Alluxio stores inodes on disk more compactly than it stores them in memory. Instead of 1KB per file, it takes about 500 bytes. To support 1 billion files stored on disk, provision an HDD or SSD with 500GB of space, and make sure the RocksDB metastore is using that disk.

alluxio-site.properties

alluxio.master.metastore.dir=/path/to/metastore

3 Conclusion

Alluxio 2.0 significantly improves metadata scalability by taking advantage of disk resources for cold metadata. This prepares Alluxio to handle larger namespace volumes as people mount more storages into Alluxio and make greater use of object storage. Stay tuned for a followup blog on the technical details of how we leverage RocksDB for storing our metadata.

Alluxio File system Metadata Memory (storage engine)

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • DevSecOps Benefits and Challenges
  • The Real Democratization of AI, and Why It Has to Be Closely Monitored
  • Spring Cloud: How To Deal With Microservice Configuration (Part 1)
  • Agile Scrum and the New Way of Work in 2023

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: