Search in Big Files Via MappedByteBuffer

This application is useful for searching in big files that don't fit well for BufferedReader,  Files.readAllLines(), Files.lines(), and Scanner.

The solution that we'll talk about here is based on Java NIO.2, MappedByteBuffer, and FileChannel. This solution opens a memory-mapped byte buffer (MappedByteBuffer) from a FileChannel on the given file. We traverse the fetched byte buffer and look for matches with the searched string (this string is converted into a byte[] and searching take place byte by byte).

For relatively small files, it is faster to load the entire file into memory (if RAM allows it). For large/huge files, it is faster to load and process the files in chunks (for example, a chunk of 5 MB). Once we have loaded a chunk, we have to count the number of occurrences of the searched string. We store the result and pass it to the next chunk of data. We repeat this until the entire file has been traversed.

Let's take a look at the core lines of this implementation (take a look at the source code bundled with this book for the complete code):

private static final long MAP_SIZE = 5242880; // 5 MB in bytes

public static long countOccurrences(Path path, String text)
                                           throws IOException {

   final byte[] texttofind = text.getBytes(StandardCharsets.UTF_8);
   long count = 0;

   try (FileChannel fileChannel = FileChannel.open(path,
                                           StandardOpenOption.READ)) {
      long position = 0;
      long length = fileChannel.size();

      while (position < length) {
         long remaining = length - position;
         long bytestomap = (long) Math.min(MAP_SIZE, remaining);

         MappedByteBuffer mbBuffer = fileChannel.map(
                          MapMode.READ_ONLY, position, bytestomap);
         long limit = mbBuffer.limit();
         long lastSpace = -1;
         long firstChar = -1;
         while (mbBuffer.hasRemaining()) {
            // code omitted for brevity available on GitHub
            ...
         }
      }
   }

   return count;
}

This solution is extremely fast because the file is read directly from the operating system's memory without having to be loaded into the JVM. The operations take place at the native level, called the operating system level. Note that this implementation works only for the UTF-8 charset, but it can be adapted for other charsets as well.

The complete application is available on GitHub.

If you enjoyed this article, then I'm sure you will love Chapter 6 from Java Coding Problem. This chapter is dedicated to file/folders manipulation in Java.

Further Reading

Comparing Files in Java