DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • Universal Implementation of BFS, DFS, Dijkstra, and A-Star Algorithms
  • Linked List in Data Structures and Algorithms
  • Queue in Data Structures and Algorithms
  • Generics in Java and Their Implementation

Trending

  • Dropwizard vs. Micronaut: Unpacking the Best Framework for Microservices
  • How To Develop a Truly Performant Mobile Application in 2025: A Case for Android
  • Creating a Web Project: Caching for Performance Optimization
  • Cosmos DB Disaster Recovery: Multi-Region Write Pitfalls and How to Evade Them
  1. DZone
  2. Data Engineering
  3. Data
  4. A Memory Efficient and Fast Byte Array Builder Implementation

A Memory Efficient and Fast Byte Array Builder Implementation

Learn how to build a fast byte array builder without the use of any Java libraries.

By 
Sutanu Dalui user avatar
Sutanu Dalui
·
Jul. 27, 15 · Tutorial
Likes (2)
Comment
Save
Tweet
Share
23.9K Views

Join the DZone community and get the full member experience.

Join For Free

In many applications we come across scenarios where we need a builder for byte array, similar to say like a StringBuilder. We seek for an option where we can go appending bytes (array), and finally get the constructed array.

Since Java does not provide any such library, most create a simple utility with at least the following behaviour.

byte[] toBytes();
ByteArrayBuilder append(byte[] bytes);

In its simplest form (Java 1.6+), an implementation can be as follows:

ByteArrayBuilder append(byte[] _bytes)
{
int offset = values.length;
values = Arrays.copyOf(values, offset+_bytes.length);
System.arraycopy(_bytes, 0, values, offset, _bytes.length);
return this;
}

byte[] toBytes() {
return values;
}

The full class is not provided, for brevity purpose.

This method is one of the various ways this can be implemented. However, in any such solutions, there will be multiple intermediate array instantiations (line 4, above). This can turn ugly with a high heap memory usage (although that should be in eden space), where big byte arrays are being created from within a loop, and result into an OOM exception.

I was specifically facing "GC overhead limit exceeded" and "Java heap space" errors, while running a test with fixed max heap size.

Can we eliminate this intermediate array instantiation? The solution I am providing here does exactly that. PLEASE NOTE: To avoid intermediate instantions, we would be utilising off-heap allocations using the (in)famous sun.misc.Unsafe.

Compared to the previous solution this produced around 3.5 times less heap usage, as tested in my laptop (Ubuntu 12.04, 8gb ram) using Java1.7u79 with no additional JVM parameters. Also the performance was slightly better as well (execution timing), though I would really consider the memory efficiency more.

package com.vortex.files.util;

import java.lang.reflect.Field;

import sun.misc.Unsafe;

public class ByteArrayBuilder implements ArrayBuilder<byte[]> 
{
private static Unsafe UNSAFE;
static
{
try {
Field f = Unsafe.class.getDeclaredField("theUnsafe");
f.setAccessible(true);
UNSAFE = (Unsafe) f.get(null);
} catch (Exception e) {
throw new ExceptionInInitializerError("sun.misc.Unsafe not instantiated ["+e.toString()+"]");
}

}

private long _address = -1;
private int byteSize = 0;

/* (non-Javadoc)
 * @see com.vortex.files.util.ArrayBuilder#toArray()
 */
@Override
public byte[] toArray() {
byte[] values = new byte[byteSize];

        UNSAFE.copyMemory(null, _address,
                          values, Unsafe.ARRAY_BYTE_BASE_OFFSET,
                          byteSize);

        return values;
}

public ByteArrayBuilder()
{
appendFirst(new byte[0]);
}
public ByteArrayBuilder(byte[] bytes)
{
appendFirst(bytes);
}
/* (non-Javadoc)
 * @see com.vortex.files.util.ArrayBuilder#append(byte[])
 */
@Override
public ByteArrayBuilder append(byte[] _bytes)
{
return appendNext(_bytes);
}
private ByteArrayBuilder appendFirst(byte[] _bytes)
{
long _address2 = UNSAFE.allocateMemory(byteSize + _bytes.length);
UNSAFE.copyMemory(_bytes, Unsafe.ARRAY_BYTE_BASE_OFFSET, null, _address2, _bytes.length);
_address = _address2;
byteSize += _bytes.length;
return this;
}
private ByteArrayBuilder appendNext(byte[] _bytes)
{
long _address2 = UNSAFE.allocateMemory(byteSize + _bytes.length);
UNSAFE.copyMemory(_address, _address2, byteSize);
UNSAFE.copyMemory(_bytes, Unsafe.ARRAY_BYTE_BASE_OFFSET+byteSize, null, _address2, _bytes.length);
UNSAFE.freeMemory(_address);
_address = _address2;
byteSize += _bytes.length;
return this;
}
/* (non-Javadoc)
 * @see com.vortex.files.util.ArrayBuilder#free()
 */
@Override
public void free() {
UNSAFE.freeMemory(_address);

}
}
Data structure Memory (storage engine) Data Types Implementation

Opinions expressed by DZone contributors are their own.

Related

  • Universal Implementation of BFS, DFS, Dijkstra, and A-Star Algorithms
  • Linked List in Data Structures and Algorithms
  • Queue in Data Structures and Algorithms
  • Generics in Java and Their Implementation

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!