DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • How To Approach Java, Databases, and SQL [Video]
  • Mastering Time Series Analysis: Techniques, Models, and Strategies
  • Effortlessly Streamlining Test-Driven Development and CI Testing for Kafka Developers
  • Merge GraphQL Schemas Using Apollo Server and Koa

Trending

  • How To Approach Java, Databases, and SQL [Video]
  • Mastering Time Series Analysis: Techniques, Models, and Strategies
  • Effortlessly Streamlining Test-Driven Development and CI Testing for Kafka Developers
  • Merge GraphQL Schemas Using Apollo Server and Koa
  1. DZone
  2. Coding
  3. Java
  4. 5 Simple Binary Encoding Gotchas

5 Simple Binary Encoding Gotchas

Spending hours in frustration debugging SBE issues in your application? This post covers up to 80% of common usage issues related to simple binary encoding.

Tommy Q user avatar by
Tommy Q
·
Sep. 21, 20 · Tutorial
Like (3)
Save
Tweet
Share
6.16K Views

Join the DZone community and get the full member experience.

Join For Free

Spending hours in frustration debugging Simple Binary Encoding (SBE) issues in your application? You aren’t alone. I’ve been there before. This post hopes to alleviate some of your pains by covering up to 80% of common usage issues related to SBE (I believe).

Code examples: https://github.com/tommyqqt/sbe-gotchas.git

Let’s Recap

SBE is an ultra-fast codec commonly used in low latency financial applications such as FIX engines, pricing engines, etc. This post assumes that you are familiar with the basics.

If you are new to SBE, visit https://github.com/real-logic/simple-binary-encoding

This post refers to the specific SBE implementation in Java (version 1.19.0) developed by Real-Logic. It is not about the SBE FIX standard.

header - block fields - repeating group 1 - repeating group 2 - ... - repeating group N - var-length fields

The same structure Block Fields-Repeating Groups-Var length fields can also be nested in each repeating group.

Fields in an SBE message have to be encoded/decoded sequentially unless the limit is at the beginning of a block whose fixed-length members can be accessed randomly.

Now let’s jump to the common gotchas!

1. When Encoded Length Isn’t Encoded Length

There are times when we would want to know the encoded length of an SBE message, such as sending the message over the wire or persisting it to a file.

How would you get the encoded length of an SBE message? If we just finished encoding the message and we have the encoder on hand, isn’t it just simply calling encoder.encodedLength()? Let’s try it out.

Java
 




xxxxxxxxxx
1


 
1
final int encodedLength = nosEncoder.encodedLength();
2
final byte[] bytes = new byte[encodedLength];
3
buffer.getBytes(0, bytes);
4
final DirectBuffer readBuffer = new UnsafeBuffer(bytes);
5
wrapDecoder(headerDecoder, nosDecoder, readBuffer, 0);
6
System.out.println(nosDecoder);



Plain Text
 




xxxxxxxxxx
1


 
1
java.lang.IndexOutOfBoundsException: index=262 length=22 capacity=276


We get an exception because encoder.encodedLength() excludes the header length. The whole byte array is required to decode the message, not just the body.

Java
 




xxxxxxxxxx
1


 
1
int encodedLengthFromDecoder = headerDecoder.encodedLength() + nosDecoder.encodedLength();



How to determine the encoded length if we only have the encoded buffer? Unfortunately, the decoder has to traverse to the end of the message to get the encoded length. Or, one other way is to remember the encoded length at the time that the message was encoded and pass it along with the encoded buffer as a method parameter.

Java
 




xxxxxxxxxx
1
12


 
1
//Skip to the end
2
skipGroup(nosDecoder.allocations(), allocDec -> {
3
    skipGroup(allocDec.nestedParties(), partyDec -> {
4
               partyDec.nestedPartyDescription();
5
    });
6
    allocDec.allocDescription();
7
});
8
nosDecoder.traderDescription();
9
nosDecoder.orderDescription();
10

          
11
//decoder encoded length at end of message = actual encoded Length
12
encodedLengthFromDecoder = headerDecoder.encodedLength() + nosDecoder.encodedLength();


2. The Moving Repeating Group

One habit that we Java programmers usually adopt is that sometimes when we need to use a value returned by a method call multiple times whereby the value is supposed to stay the same between calls, we then call the method many times, instead of assigning its result to a local variable.

What happen when we try to obtain a reference to the start of a repeating group multiple times like the code below? Note that we haven’t even attempted to traverse the repeating group yet (by calling next()).

Java
 




xxxxxxxxxx
1


 
1
//At start of repeating group, print the group count and current limit
2
System.out.println("Number of allocations: " + nosDecoder.allocations().count());
3
System.out.println("Current limit: " + nosDecoder.limit());
4

          
5
//Print the group count and limit again
6
System.out.println("Number of allocations: " + nosDecoder.allocations().count());
7
System.out.println("Current limit: " + nosDecoder.limit());


Common sense says we should get the sames count and limit both times, but it doesn’t work that way for SBE.

Plain Text
 




xxxxxxxxxx
1


 
1
Number of allocations: 2
2
Current limit: 30
3
Number of allocations: 20291
4
Current limit: 34



3. Mutating Var Length Field

What if there is a field whose value want to mutate after we have encoded the message? If we know which field we intended to backtrack later in advance, remember the limit just before encoding it, then use the limit to backtrack later.

Java
 




xxxxxxxxxx
1
10


 
1
//I want to change trader description later so remember the limit here
2
final int limit = nosEncoder.limit();
3
nosEncoder.traderDescription("TRADER-1");
4
nosEncoder.orderDescription("ORDER DESC");
5

          
6
nosEncoder.limit(limit);
7
nosEncoder.traderDescription("TRADER-00001");
8

          
9
//Everything subsequent to the above needs to be encoded again
10
nosEncoder.orderDescription("ORDER DESC");


Unless the field is a fixed length field, every field subsequent to the mutated field needs to be encoded again.

4. The Semi-Forbidden Schema Evolution

Suppose “orderDescription” is a new var-length field that has just been added to the end of the schema like this:

XML
 




xxxxxxxxxx
1
17


 
1
<sbe:message name="NewOrderSingle" id="0001" description="Example NewOrderSingle">
2
        <field name="orderId" id="11" type="fixedStringEncoding16"/>
3
        <field name="tradeDate" id="75" type="uint16"/>
4
        <group name="allocations" id="78" dimensionType="groupSizeEncoding">
5
            <field name="allocAccount" id="79" type="fixedStringEncoding16"/>
6
            <field name="allocQty" id="80" type="double"/>
7
            <group name="nestedParties" id="539" dimensionType="groupSizeEncoding">
8
                <field name="nestedPartyID" id="524" type="fixedStringEncoding16"/>
9
                <field name="nestedPartyRole" id="538" type="fixedStringEncoding16"/>
10
                <data name="nestedPartyDescription" id="6051" type="varStringEncoding"/>
11
            </group>
12
            <data name="allocDescription" id="6052" type="varStringEncoding"/>
13
        </group>
14
        <data name="traderDescription" id="6053" type="varStringEncoding"/>
15
        <!-- new var length field -->
16
        <data name="orderDescription" id="6054" type="varStringEncoding"/>
17
</sbe:message>


If we are not using that field now, can we not have to change our code to encode/decode that new field? It’s at the end of the message anyway and surely regression test doesn’t pick up anything!

What happens when the code below runs?

Java
 




xxxxxxxxxx
1
15


 
1
encodeOrder(nosEncoder, "ORDER-001", 20200701,
2
                "TRADER-0123456789",
3
                null,
4
                buffer);
5
//Flip the buffer to decoder
6
wrapDecoder(headerDecoder, nosDecoder, buffer, 0);
7
System.out.println(nosDecoder);
8

          
9
encodeOrder(nosEncoder, "ORDER-002", 20200701,
10
                "TRADER-0001", //Longer trader desc
11
                null,
12
                buffer);
13
//Flip the buffer to decoder
14
wrapDecoder(headerDecoder, nosDecoder, buffer, 0);
15
System.out.println(nosDecoder);


It explodes.

Plain Text
 




xxxxxxxxxx
1


 
1
java.lang.IndexOutOfBoundsException: index=265 length=926299444 capacity=384


Code that uses SBE also tends to reuse the buffers to reduce allocations. Even though we don’t care about the last field, the buffer may contains some bytes from the previous message that encroaches on the new field when we encode the new message.

5. Debugging and Testing with Base64 Encoding

How to troubleshoot SBE issues in production?

A couple of ways:

  1. Replay the SBE messages in your test harness (if you employ event sourcing pattern)
  2. Trawl through log files for problematic SBE messages

Let’s talk about (2). Below is a string representation of an SBE message.

Plain Text
 




xxxxxxxxxx
1


 
1
[NewOrderSingle](sbeTemplateId=1|sbeSchemaId=1|sbeSchemaVersion=1|sbeBlockLength=18):orderId=ORDER-001|tradeDate=15613|allocations=[(allocAccount=ACCOUNT-1|allocQty=100.0|nestedParties=[(nestedPartyID=Party-1|nestedPartyRole=|nestedPartyDescription='Party-1')]|allocDescription='ALLOCATION WITH ACCOUNT ACCOUNT-1'),(allocAccount=ACCOUNT-2|allocQty=200.0|nestedParties=[(nestedPartyID=Party-2|nestedPartyRole=|nestedPartyDescription='Party-2')]|allocDescription='ALLOCATION WITH ACCOUNT ACCOUNT-2')]|traderDescription='TRADER-0123456789'|orderDescription=''


Besides eyeballing it, is there a way to turn text into SBE bytes (i.e. similar to Protobuf TextFormat parser)? You can write one your own or look hard enough for SBE parsers on the internet. The only problem is that SBE string representation is somewhat arbitrary. It is not a well-defined language like JSON. SBE parsers can stop working if there is an extra pipe or parenthesis somewhere in the text.

Java 8 ‘s Base64 encoding comes to the rescue. We can print the SBE’s bytes as a string anywhere, be it in the log files or even in Junit test cases, and easily reconstruct the bytes later on. No longer need to worry about storing SBE messages as binary files. Yay!

Java
 




xxxxxxxxxx
1
13


 
1

          
2
final int encodedLength = headerEncoder.encodedLength() + nosEncoder.encodedLength();
3
final byte[] bytes = new byte[encodedLength];
4
buffer.getBytes(0, bytes);
5

          
6
final String base64EncStr = Base64.getEncoder().encodeToString(bytes);
7
System.out.println(base64EncStr);
8

          
9
final byte[] decoderBytes = Base64.getDecoder().decode(base64EncStr);
10
final DirectBuffer decoderBuffer = new UnsafeBuffer(decoderBytes);
11
wrapDecoder(headerDecoder, nosDecoder, decoderBuffer, 0);
12
final String decoderToString = nosDecoder.toString();
13
System.out.println(decoderToString);



Plain Text
 




xxxxxxxxxx
1


 
1
SBE Base64 encoding string:
2
EgABAAEAAQBPUkRFUklELTAwMQAAAAAA/TwYAAIAQUNDT1VOVC0xAAAAAAAAAAAAAAAAAFlAIAABAFBhcnR5LTEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABwAAAFBhcnR5LTEhAAAAQUxMT0NBVElPTiBXSVRIIEFDQ09VTlQgQUNDT1VOVC0xQUNDT1VOVC0yAAAAAAAAAAAAAAAAAGlAIAABAFBhcnR5LTIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABwAAAFBhcnR5LTIhAAAAQUxMT0NBVElPTiBXSVRIIEFDQ09VTlQgQUNDT1VOVC0yCAAAAFRSQURFUi0xFgAAAERVTU1ZIE5FVyBPUkRFUiBTSU5HTEU=



Best of luck on your SBE adventure and don’t forget to share the tips if you think they make your life easier!

Specification by example Plain text Java (programming language)

Published at DZone with permission of Tommy Q. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • How To Approach Java, Databases, and SQL [Video]
  • Mastering Time Series Analysis: Techniques, Models, and Strategies
  • Effortlessly Streamlining Test-Driven Development and CI Testing for Kafka Developers
  • Merge GraphQL Schemas Using Apollo Server and Koa

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: