5 Simple Binary Encoding Gotchas
Spending hours in frustration debugging SBE issues in your application? This post covers up to 80% of common usage issues related to simple binary encoding.
Join the DZone community and get the full member experience.
Join For FreeSpending hours in frustration debugging Simple Binary Encoding (SBE) issues in your application? You aren’t alone. I’ve been there before. This post hopes to alleviate some of your pains by covering up to 80% of common usage issues related to SBE (I believe).
Code examples: https://github.com/tommyqqt/sbe-gotchas.git
Let’s Recap
SBE is an ultra-fast codec commonly used in low latency financial applications such as FIX engines, pricing engines, etc. This post assumes that you are familiar with the basics.
If you are new to SBE, visit https://github.com/real-logic/simple-binary-encoding
This post refers to the specific SBE implementation in Java (version 1.19.0) developed by Real-Logic. It is not about the SBE FIX standard.
The same structure Block Fields-Repeating Groups-Var length fields can also be nested in each repeating group.
Fields in an SBE message have to be encoded/decoded sequentially unless the limit is at the beginning of a block whose fixed-length members can be accessed randomly.
Now let’s jump to the common gotchas!
1. When Encoded Length Isn’t Encoded Length
There are times when we would want to know the encoded length of an SBE message, such as sending the message over the wire or persisting it to a file.
How would you get the encoded length of an SBE message? If we just finished encoding the message and we have the encoder on hand, isn’t it just simply calling encoder.encodedLength()? Let’s try it out.
Java
xxxxxxxxxx
1
1
final int encodedLength = nosEncoder.encodedLength();
2
final byte[] bytes = new byte[encodedLength];
3
buffer.getBytes(0, bytes);
4
final DirectBuffer readBuffer = new UnsafeBuffer(bytes);
5
wrapDecoder(headerDecoder, nosDecoder, readBuffer, 0);
6
System.out.println(nosDecoder);
Plain Text
xxxxxxxxxx
1
1
java.lang.IndexOutOfBoundsException: index=262 length=22 capacity=276
We get an exception because encoder.encodedLength() excludes the header length. The whole byte array is required to decode the message, not just the body.
Java
xxxxxxxxxx
1
1
int encodedLengthFromDecoder = headerDecoder.encodedLength() + nosDecoder.encodedLength();
How to determine the encoded length if we only have the encoded buffer? Unfortunately, the decoder has to traverse to the end of the message to get the encoded length. Or, one other way is to remember the encoded length at the time that the message was encoded and pass it along with the encoded buffer as a method parameter.
Java
xxxxxxxxxx
1
12
1
//Skip to the end
2
skipGroup(nosDecoder.allocations(), allocDec -> {
3
skipGroup(allocDec.nestedParties(), partyDec -> {
4
partyDec.nestedPartyDescription();
5
});
6
allocDec.allocDescription();
7
});
8
nosDecoder.traderDescription();
9
nosDecoder.orderDescription();
10
11
//decoder encoded length at end of message = actual encoded Length
12
encodedLengthFromDecoder = headerDecoder.encodedLength() + nosDecoder.encodedLength();
2. The Moving Repeating Group
One habit that we Java programmers usually adopt is that sometimes when we need to use a value returned by a method call multiple times whereby the value is supposed to stay the same between calls, we then call the method many times, instead of assigning its result to a local variable.
What happen when we try to obtain a reference to the start of a repeating group multiple times like the code below? Note that we haven’t even attempted to traverse the repeating group yet (by calling next()).
Java
xxxxxxxxxx
1
1
//At start of repeating group, print the group count and current limit
2
System.out.println("Number of allocations: " + nosDecoder.allocations().count());
3
System.out.println("Current limit: " + nosDecoder.limit());
4
5
//Print the group count and limit again
6
System.out.println("Number of allocations: " + nosDecoder.allocations().count());
7
System.out.println("Current limit: " + nosDecoder.limit());
Common sense says we should get the sames count and limit both times, but it doesn’t work that way for SBE.
Plain Text
xxxxxxxxxx
1
1
Number of allocations: 2
2
Current limit: 30
3
Number of allocations: 20291
4
Current limit: 34
3. Mutating Var Length Field
What if there is a field whose value want to mutate after we have encoded the message? If we know which field we intended to backtrack later in advance, remember the limit just before encoding it, then use the limit to backtrack later.
Java
xxxxxxxxxx
1
10
1
//I want to change trader description later so remember the limit here
2
final int limit = nosEncoder.limit();
3
nosEncoder.traderDescription("TRADER-1");
4
nosEncoder.orderDescription("ORDER DESC");
5
6
nosEncoder.limit(limit);
7
nosEncoder.traderDescription("TRADER-00001");
8
9
//Everything subsequent to the above needs to be encoded again
10
nosEncoder.orderDescription("ORDER DESC");
Unless the field is a fixed length field, every field subsequent to the mutated field needs to be encoded again.
4. The Semi-Forbidden Schema Evolution
Suppose “orderDescription” is a new var-length field that has just been added to the end of the schema like this:
XML
xxxxxxxxxx
1
17
1
<sbe:message name="NewOrderSingle" id="0001" description="Example NewOrderSingle">
2
<field name="orderId" id="11" type="fixedStringEncoding16"/>
3
<field name="tradeDate" id="75" type="uint16"/>
4
<group name="allocations" id="78" dimensionType="groupSizeEncoding">
5
<field name="allocAccount" id="79" type="fixedStringEncoding16"/>
6
<field name="allocQty" id="80" type="double"/>
7
<group name="nestedParties" id="539" dimensionType="groupSizeEncoding">
8
<field name="nestedPartyID" id="524" type="fixedStringEncoding16"/>
9
<field name="nestedPartyRole" id="538" type="fixedStringEncoding16"/>
10
<data name="nestedPartyDescription" id="6051" type="varStringEncoding"/>
11
</group>
12
<data name="allocDescription" id="6052" type="varStringEncoding"/>
13
</group>
14
<data name="traderDescription" id="6053" type="varStringEncoding"/>
15
<!-- new var length field -->
16
<data name="orderDescription" id="6054" type="varStringEncoding"/>
17
</sbe:message>
If we are not using that field now, can we not have to change our code to encode/decode that new field? It’s at the end of the message anyway and surely regression test doesn’t pick up anything!
What happens when the code below runs?
Java
xxxxxxxxxx
1
15
1
encodeOrder(nosEncoder, "ORDER-001", 20200701,
2
"TRADER-0123456789",
3
null,
4
buffer);
5
//Flip the buffer to decoder
6
wrapDecoder(headerDecoder, nosDecoder, buffer, 0);
7
System.out.println(nosDecoder);
8
9
encodeOrder(nosEncoder, "ORDER-002", 20200701,
10
"TRADER-0001", //Longer trader desc
11
null,
12
buffer);
13
//Flip the buffer to decoder
14
wrapDecoder(headerDecoder, nosDecoder, buffer, 0);
15
System.out.println(nosDecoder);
It explodes.
Plain Text
xxxxxxxxxx
1
1
java.lang.IndexOutOfBoundsException: index=265 length=926299444 capacity=384
Code that uses SBE also tends to reuse the buffers to reduce allocations. Even though we don’t care about the last field, the buffer may contains some bytes from the previous message that encroaches on the new field when we encode the new message.
5. Debugging and Testing with Base64 Encoding
How to troubleshoot SBE issues in production?
A couple of ways:
- Replay the SBE messages in your test harness (if you employ event sourcing pattern)
- Trawl through log files for problematic SBE messages
Let’s talk about (2). Below is a string representation of an SBE message.
Plain Text
xxxxxxxxxx
1
1
[NewOrderSingle](sbeTemplateId=1|sbeSchemaId=1|sbeSchemaVersion=1|sbeBlockLength=18):orderId=ORDER-001|tradeDate=15613|allocations=[(allocAccount=ACCOUNT-1|allocQty=100.0|nestedParties=[(nestedPartyID=Party-1|nestedPartyRole=|nestedPartyDescription='Party-1')]|allocDescription='ALLOCATION WITH ACCOUNT ACCOUNT-1'),(allocAccount=ACCOUNT-2|allocQty=200.0|nestedParties=[(nestedPartyID=Party-2|nestedPartyRole=|nestedPartyDescription='Party-2')]|allocDescription='ALLOCATION WITH ACCOUNT ACCOUNT-2')]|traderDescription='TRADER-0123456789'|orderDescription=''
Besides eyeballing it, is there a way to turn text into SBE bytes (i.e. similar to Protobuf TextFormat parser)? You can write one your own or look hard enough for SBE parsers on the internet. The only problem is that SBE string representation is somewhat arbitrary. It is not a well-defined language like JSON. SBE parsers can stop working if there is an extra pipe or parenthesis somewhere in the text.
Java 8 ‘s Base64 encoding comes to the rescue. We can print the SBE’s bytes as a string anywhere, be it in the log files or even in Junit test cases, and easily reconstruct the bytes later on. No longer need to worry about storing SBE messages as binary files. Yay!
Java
xxxxxxxxxx
1
13
1
2
final int encodedLength = headerEncoder.encodedLength() + nosEncoder.encodedLength();
3
final byte[] bytes = new byte[encodedLength];
4
buffer.getBytes(0, bytes);
5
6
final String base64EncStr = Base64.getEncoder().encodeToString(bytes);
7
System.out.println(base64EncStr);
8
9
final byte[] decoderBytes = Base64.getDecoder().decode(base64EncStr);
10
final DirectBuffer decoderBuffer = new UnsafeBuffer(decoderBytes);
11
wrapDecoder(headerDecoder, nosDecoder, decoderBuffer, 0);
12
final String decoderToString = nosDecoder.toString();
13
System.out.println(decoderToString);
Plain Text
xxxxxxxxxx
1
1
SBE Base64 encoding string:
2
EgABAAEAAQBPUkRFUklELTAwMQAAAAAA/TwYAAIAQUNDT1VOVC0xAAAAAAAAAAAAAAAAAFlAIAABAFBhcnR5LTEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABwAAAFBhcnR5LTEhAAAAQUxMT0NBVElPTiBXSVRIIEFDQ09VTlQgQUNDT1VOVC0xQUNDT1VOVC0yAAAAAAAAAAAAAAAAAGlAIAABAFBhcnR5LTIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABwAAAFBhcnR5LTIhAAAAQUxMT0NBVElPTiBXSVRIIEFDQ09VTlQgQUNDT1VOVC0yCAAAAFRSQURFUi0xFgAAAERVTU1ZIE5FVyBPUkRFUiBTSU5HTEU=
Best of luck on your SBE adventure and don’t forget to share the tips if you think they make your life easier!
Topics:
java,
low latency message processing,
performance
Published at DZone with permission of Tommy Q. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments