Top 5 Challenges in Performance Testing of Low Latency Electronic Trading System

A Trading system offers a sophisticated platform to trade securities. The financial institutions enable investors/traders to open, execute, close, and manage market positions in a secure way through their trading system. Most trading systems nowadays offer automated trading controls (algorithmic trades) that help investors in making the right investment strategy decisions quickly, considering all risk possibilities and yielding the best benefits. The trading system is expected to handle high trade volumes ensuring low latency throughput. Examples of the popular trading system include TradeStation, TD Ameritrade, etc.

With the emergence of FIX (Financial Information Exchange) protocol standardization and the exponential growth of high-speed electronic algorithmic trading systems, there is a strong focus on the technology stack adopted by the trading systems to meet the ever-changing needs of clients. The clients, in general, includes various market segments, including professional clients, institutional clients, and interbank clients. Even the trade execution venues face a similar technology challenge to perform quick order matching, notifying the trade-to-market data distributor in a few milliseconds. For example, NASDAQ offers a fully automated market.

Performance testing of trading systems is highly important as the system behavior needs to be validated for stressful market conditions and handling high trade volumes. As the market is moving really quickly, the high-speed algorithmic trading system needs to get the market data, decide on the trading strategy, and send the order to the venue with microseconds latency.  A highly effective trading system offers high throughput (trade volumes) and low latency. In the trading world, time is money. The latency exhibited by the system for handling the trades affects the trader's experience. Hence, validating the trading system to ensure compliance with the Performance NFRs (Non-Functional Requirements) becomes important.

Performance benchmarking of the trading systems for capturing various metrics like peak order volumes handled, 95% percentile latency, hardware utilization levels, system uptime, etc., is essential. Performance testing also helps uncover trading system providers’ readiness to handle real market issues and the time taken for resolution. But performance testing of the trading systems comes with various challenges:

1. End-to-End Integrated Performance Test Environment 

Low-latency electronic trading applications are highly complex in nature, with various subsystems, interfaces, gateways, message brokers, market data appliances, third-party applications, various integrations with venues, etc. Setting up the required upstream and downstream systems to bring production realistic setup is highly challenging. Validating the application performance against various exchange protocols requires exchange simulators to be built and configured in the test lab. Setting up sophisticated market data replay appliances in production might be mandatory to meet regulatory compliance instead of using simple production feeds replay solutions. Replicating the connectivity for the front office, middle office, and back office systems along with all interfaces and subsystems in the controlled environment replicating the production environment is almost impossible. Isolated performance tests of components and subsystems are performed where the entire system setup becomes challenging. An appropriate performance testing strategy needs to be devised to extrapolate the performance KPIs measured from lower environments to production capacity.

2. Realistic Production Workload Simulation

Quantitative and Qualitative simulation of realistic workload in performance test environment requires a good amount of analysis of production workloads. As in any application, production peaks for each trading application/component/subsystem of the end-to-end order management system (OMS) need to be studied thoroughly to finalize the Performance NFRs. Simulation of OMS production workload in the test environment is not simple as in the case of a typical web application. Beyond identifying peak order volumes and types of orders (like New/Mod/Cancel/RFQ or market/limit order), additional integrities are to be considered for each type of application/component. For example, in the case of algo-trading application, parent-child order ratio, top smart order routing strategies, realistic mix, etc., need to be considered, which could be very different from the workload considerations of a market-making application or risk validation application.

3. Strict Compliance Against Regulatory Policy Requirements

Electronic Trading systems need to strictly comply with Regulatory performance test requirements (MiFID II). The regulatory policies dictate the type of performance test to be executed (say, load test, capacity test, throughput test, breakpoint test, system behavioral checks, etc.) and documentation of evidence/results. In general, all trading systems should abide by country-specific regulatory compliance requirements. The regulatory requirements should be studied, and required tests need to be carried out. For example, the MiFID II performance testing requirement states, “trading system should have sufficient capacity to perform their functions without systems failures, outages or errors in matching transactions at least at twice the highest number of messages per second recorded on that system, during the previous five years”. Any non-conformances, like the inability to test with requirement market data replay rates, etc., need to be strictly documented for auditing purposes.

4. Performance Test Tooling/Framework

The use of open-source tools like Apache JMeter or Gatling is useful for load-testing APIs or web applications. Most of the market available performance testing tools are not a right fit for carrying out a performance testing for an order management system that uses protocols like FIX or other binary protocols like OUCH, Arrowhead, Millennium, OCG, OMEX, EURONEXT, T7, etc. to communicate with the exchange. Though there are few FIX load testing tools, like Spirent MarketStress, FIXYL, etc., available in the market, developing an in-house performance testing framework with the capability to simulate the order flow would be beneficial. This will bring convenience in implementing custom requirements. But this requires dedicated skilled resources to build and maintain the tools. The tool not only should facilitate generating millions of different types of orders but also should have the capability to simulate the exchange behavior by sending acknowledgments and executing the orders as per the requirement (Open/Reject/Part fill/Full fill/Multi-Part fill/Multi-Full fill, etc.). The tool should have the capability to report the throughput (order processing rate) and latency (round-trip and one-side) metrics. Without having sophisticated observability tools to understand the application behavior (JVM usage, GC behavior, costly methods, etc.) and infrastructure health, mere performance testing is not useful. Hence, ensuring the availability of the right set of performance monitoring/APM tools is essential. Additional tools like Pico's Corvil would be essential while testing trading systems where component-level latency calculations become vital.

5. Strong Domain Expertise

Generally, Performance Engineers don’t necessarily need strong domain knowledge though they are expected to understand the end user usage scenarios and usage patterns to simulate realistic user behavior. But in the case of the capital market domain, good domain knowledge becomes essential. Domain understanding and knowledge of e-trading systems become super important to the device's right performance test strategy and troubleshooting the issues in the order flows. For example, the order structure to be used for market order versus limit order, and if the limit order is used, how to fetch the latest market rates for the ticker symbols used in the test suite should be known by the performance engineer. Expertise in FIX protocol is a pre-requisite for any Performance engineer to ensure the right order structure (with the right tags) template is used for injecting the orders into the system.

These are some of the key challenges for Performance Testing of trading systems.

With Agile software development becoming the de facto standard and increased DevOps adoption paving the way for weekly (or even less) software releases, carrying out performance tests integrated within the delivery pipeline is essential to benchmark and validate the trading system performance before the production release.

Reactive performance testing thoughts at the end of SDLC or carrying out tests for the sake of complying with regulatory requirements will not be beneficial where the performance of the trading system is critical for the business win.

Achieving high performance is possible only through Proactive Performance Management strategies. System performance should be made everybody’s goal where collaboration between various persona across the SDLC life cycle is essential. Everyone should accept and work towards the common goal of achieving high-performance trading systems to enhance the client experience.