DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Because the DevOps movement has redefined engineering responsibilities, SREs now have to become stewards of observability strategy.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Trending

  • Advancing Robot Vision and Control
  • When Airflow Tasks Get Stuck in Queued: A Real-World Debugging Story
  • Simpler Data Transfer Objects With Java Records
  • Navigating and Modernizing Legacy Codebases: A Developer's Guide to AI-Assisted Code Understanding

Search in Big Files Via MappedByteBuffer

Learn more about search in big files via MappedByteBuffer.

By 
Anghel Leonard user avatar
Anghel Leonard
DZone Core CORE ·
Dec. 04, 19 · Tutorial
Likes (8)
Comment
Save
Tweet
Share
14.4K Views

Join the DZone community and get the full member experience.

Join For Free

Coffee and saucer

Learn more about search in big files via MappedByteBuffer.

This application is useful for searching in big files that don't fit well for BufferedReader,  Files.readAllLines(), Files.lines(), and Scanner.

The solution that we'll talk about here is based on Java NIO.2, MappedByteBuffer, and FileChannel. This solution opens a memory-mapped byte buffer (MappedByteBuffer) from a FileChannel on the given file. We traverse the fetched byte buffer and look for matches with the searched string (this string is converted into a byte[] and searching take place byte by byte).

For relatively small files, it is faster to load the entire file into memory (if RAM allows it). For large/huge files, it is faster to load and process the files in chunks (for example, a chunk of 5 MB). Once we have loaded a chunk, we have to count the number of occurrences of the searched string. We store the result and pass it to the next chunk of data. We repeat this until the entire file has been traversed.

Let's take a look at the core lines of this implementation (take a look at the source code bundled with this book for the complete code):

private static final long MAP_SIZE = 5242880; // 5 MB in bytes

public static long countOccurrences(Path path, String text)
                                           throws IOException {

   final byte[] texttofind = text.getBytes(StandardCharsets.UTF_8);
   long count = 0;

   try (FileChannel fileChannel = FileChannel.open(path,
                                           StandardOpenOption.READ)) {
      long position = 0;
      long length = fileChannel.size();

      while (position < length) {
         long remaining = length - position;
         long bytestomap = (long) Math.min(MAP_SIZE, remaining);

         MappedByteBuffer mbBuffer = fileChannel.map(
                          MapMode.READ_ONLY, position, bytestomap);
         long limit = mbBuffer.limit();
         long lastSpace = -1;
         long firstChar = -1;
         while (mbBuffer.hasRemaining()) {
            // code omitted for brevity available on GitHub
            ...
         }
      }
   }

   return count;
}


This solution is extremely fast because the file is read directly from the operating system's memory without having to be loaded into the JVM. The operations take place at the native level, called the operating system level. Note that this implementation works only for the UTF-8 charset, but it can be adapted for other charsets as well.

The complete application is available on GitHub.

If you enjoyed this article, then I'm sure you will love Chapter 6 from Java Coding Problem. This chapter is dedicated to file/folders manipulation in Java.

Further Reading

Comparing Files in Java

Opinions expressed by DZone contributors are their own.

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!