CQRS and Event Sourcing Intro For Developers

In this post, we focus on the basics concepts of CQRS and event sourcing, augmenting these explanation with visualizations and examples.

Wojciech Suwała

Robert Witkowski

May. 28, 19 · Tutorial

Like (21)

Save

32.6K Views

We live in a world of dynamically changing technologies. New ways of architecturing our solutions, new frameworks and libraries seem to appear on almost a daily basis. But good software engineering is not about fancy frameworks and solutions aggressively promoted by their vendors. It is not about doing something because Netflix or Google did it. It is about taking well-thought-out decisions based on facts and knowledge. That’s why it is important to be familiar basic architectural concepts like CQRS. It is one of the tools we use in our software house every day. We mentioned CQRS in the article which is part of the series about Microservices on .NET Core, but it was presented from technical perspective and here we want to focus on basics concepts and their explanation with visualizations and examples.

What Is CQRS?

Command-Query Responsibility Segregation is a pattern that tells us to separate operations that mutate data from the ones that query it. It is derived from Command Query Separation (CQS) principle devised by Bertrand Mayer – author of Eiffel.

CQS states that there can be only two kinds of methods on a class: the ones that mutate state and return void and the ones that return state but do not change it.

Greg Young is the person responsible for naming this pattern CQRS and popularizing it. If you search the internet for CQRS you will find many excellent posts and videos made by Greg. For example, you can find excellent and very simple explanation of CQRS pattern in this post.

We want to show an example from the insurance domain – PolicyService, which is service responsible for managing insurance policies. Below is a snippet with an interface before applying CQRS. All methods (write and read) are in one class.

interface PolicyService {
    void ConvertOfferToPolicy(ConvertOfferRequest convertReq);
    PolicyDetailsDto GetPolicy(long id);
    void AnnexPolicy(AnnexRequestDto annexReq);
    List SearchPolicies(PolicySearchFilter filter);
    void TerminatePolicy(TerminatePolicyRequest terminateReq);
    void ChangePayer(ChangePayerRequest req);
    List FindPoliciesToRenew(RenewFilter filter);
}

Before applying CQRS

If we apply the CQRS pattern to this case, we get two separated classes, better fulfilling the SRP principle.

interface PolicyComandService {
    void ConvertOfferToPolicy(ConvertOfferRequest convertReq);
    void AnnexPolicy(AnnexRequestDto annexReq);
    void TerminatePolicy(TerminatePolicyRequest terminateReq);
    void ChangePayer(ChangePayerRequest req);
}

interface PolicyQueryService {
    PolicyDetailsDto GetPolicy(long id);
    List SearchPolicies(PolicySearchFilter filter);
    List FindPoliciesToRenew(RenewFilter filter);  
}

After applying CQRS

That’s all. If you do this, you can show off in front of your colleagues over beer that you’re using CQRS. But it is just a mechanical transformation, which is the first step in applying CQRS. What seems like a simple transformation actually has great consequences and opens new possibilities, which we will explore further in this post.

What CQRS Enables

Most of the time data that needs to change state is different in form or quantity than data that's needed by the user to make a decision. Having the same model to handle both queries and commands results in a model that is bloated with things needed by only one type of operation; also, model complexity is increased and aggregate size is usually bigger.

CQRS enables us to have different models for mutating state and different models to support queries. Usually, write operations are less frequent than reads. Having separate models and separated database engines allows us to independently scale the query side and to better handle concurrent access as reads no longer block writes or the command side (in the opposite case). Having separate models for commands and queries lets us assign these responsibilities to different teams with different skills. For example, you can assign the command side to highly skilled OOP developers while the query side can be implemented by SQL developers. CQRS let you scale your team and lets the best devs focus on the core stuff.

Is CQRS an Architecture?

People often get this wrong. CQRS is not a top level/system level architecture. Example of architecture styles are layered, ports, and adapters. CQRS is a pattern you apply on “the inside” of your service/application and you may apply it only to a portion of your service.

Example of Implementation – Evolutionary Approach

There are many ways to implement CQRS. They have different consequences and complexity, and the applicability of such solution depends on your system context. You need a different approach if you are Netflix with 150 millions of users and a different solution will work for your typical enterprise app with just hundreds of users.

In our opinion, especially when dealing with existing (legacy) projects, the best approach is tackling CQRS evolutionarily. Below we will try to describe some of the “levels of advancement” of CQRS implementation in an application. But remember, often you do not need to use advanced patterns and solutions. Venkat Subramaniam wrote a great tweet about this:

No good programmer retires thinking: I wish I had implemented more design patterns. Let’s solve real problems using simple, pragmatic solutions.

Not always (in fact, quite rarely) your application needs to reach the highest level, because this level may result in a very complex technical solution without satisfying benefits.

No CQRS

We begin with solution that is not using CQRS.

The UI (through a controllers layer) uses a service facade layer, which coordinates business operations performed by the domain model. This model is stored in a relational database. In our example, there is the PolicyService class (Java, C#), which is responsible for handling all business methods related to the policy.

We use one model for reads and writes. When performing business operations, we use the same classes that we use when searching. This may result in your domain model having attributes needed only for search or, in a worse case, you may be forced to design your domain model to be able to query it easier.

If you want to learn more, check our examples, in Java or in C#.

Separate Model Commands and Queries

In this example we want to show the code that developers use in separated models for the write side and for the read side.

In this example, a mediator pattern is used. The role of the mediator is to ensure the delivery of a Command or Query to its Handler. The mediator receives a Command/Query, which is nothing more than a message describing intent, and passes this to a Handler which is then responsible for using the domain model to perform the expected behavior. This process can therefore be looked upon as a call to the Service Layer – where the Bus does the plumbing of the message in-between. In Java example, we created the Bus class, which is the implementation of this pattern. Registry is responsible for linking handlers with commands/queries. In C# example, we use the MediatR library that does all of this for us. You can read more about MediatR in our other article.

Now that we have separate entry points for commands and queries we can introduce different models to handle it. NoCQRS solution used RDBMS and ORM – the typical stack in an enterprise app. Without changes to our stack we can start using CQRS. With this setup we can use our domain model as a command model. This model gets simplified: some associations used only for reads are no longer necessary, and some fields are no longer necessary. On the query model, we can define views in a database and map it with ORM. Alternatively, for the query model, we can stop using the heavy-weight ORM and replace it with plain old JDBC Templates or JOOQ in Java, or Dapper in .NET. If you want to avoid complex queries that define views in databases you can take the next step and replace views with tables that will be designed to handle queries. These tables will have simple structure with data mapping 1:1 to what the user sees on the screen and what user needs to search for. Adding such tables instead of database views removes the burden of writing complex queries and opens new possibilities for scaling your solution, but it requires that you somehow keep your domain command model “in-sync” with your query model tables.

This can be done:

synchronously in the same transaction using something like Application Events in Spring (example) or using Domain Events.
synchronously in the same transaction in the command handler.
asynchronously using some kind of in-memory event bus, resulting in eventual consistency.
asynchronously using some kind of queue middleware like RabbitMQ, resulting in eventual consistency.

What are best practices for query model building like this?

You should build one table/view per screen/widget (screen fragment).
The relation between tables should be a model like relations between screen elements.
“View tables” contain columns per each field displayed on the screen.
The read model should not do any computation, calculate data in command model, and update the read model instead.
The read model should store precomputed data.
Last but not least: don’t be afraid of duplicates.

The choice of approach to take for synchronization between the command model and query model depends on many criteria. Even with database views you can achieve great results as you can scale your database with read-only replicas used only for querying with the views you have created. Having separate tables simplifies reads as you do not have to write complex SQL anymore, but you have to write code for updating query models yourself. There is no magical framework that will do this for you. The number of read models related to a given command model part is also a decision factor. If you have 2-3 query models for an aggregate you can safely call all updaters in your command handler. It won’t affect performance, but if you have tens of it, then you may consider running it asynchronously outside of the transaction that updates your aggregate. In that case you must check if eventual consistency is allowed. This is more of a business decision than a technical one and must be discussed with business users. Having separate query tables is a good step to taking our CQRS solution to next level.

If you want to learn more, check our examples, in Java or in C#.

Separate Storage Engines

In this approach, we use different storage engines for query models and for command models, for example:

ElasticSearch for the queries side, JPA for the commands side.
ElasticSearch for the queries side, DocumentDb for the commands side.
DocumentDb for the queries side, storing aggregates as JSON in RDBMS for the commands side.

Each command handler should emit events with information about what happened. An event is a named object that represents some change which happened in a given object. Events should provide information about the data that changed during the business operation. Events are part of a domain. In our example, we have some events regarding the insurance policy – PolicyCreated, PolicyAnnexed , PolicyTerminated, PolicyAnnexCancelled (Java example, C# example).

On the read side we created event handlers (methods that execute whenever a specific type of event comes in), which are responsible for projection creation. These event handlers perform CRUD operations upon the persistent read model (Java example, C# example).

What is projection? Projecting is a process of converting (or aggregating) a stream of events into a structural representation. Projection is a converted (or aggregated) stream of events into a structural representation. This can be called by many names: persistent read model, query model, or view.

With this approach we can apply different tools to do queries and state updates. This way we can achieve better performance and scalability, but at the expense of complexity. In typical business systems, the vast majority of operations performed in the system will use the read side/query model. This element should be prepared for a much higher load, it should be scalable and allow us to build complex queries that allow for advanced search.

With this approach we will have to deal with eventual consistency as distributed transactions between various data sources are performance killers and most NoSQL database do not support it.

You can take this example one step further and make query handlers in a separate microservice and command handlers in another separate services. Now you can independently scale your query and state change sides.

CQRS With Event Sourcing (CQRS-ES)

The next step is changing the command side to use Event Sourcing. The architecture of this version is very similar to the above (when we use separate storage engines).

The key difference is in the command model. Instead of RDBMS and ORM, we use an Event Store as the means of achieving persistent storage. Instead of saving only the actual state of object, we save the stream of events. This pattern for managing state is named Event Sourcing.

Instead of keeping only the current state of the system by altering the previous state, we append events (changes) to a sequential list of past events (changes). This way we not only know the current state of the system but we can easily track how we reached this state.

The below example shows different state management methods based on the football match domain.

Traditional state management

The above diagram shows the traditional state management for the Game object. We have information about the result of the match and when the game started/ended. Of course, we could model additional information here, such as a list of goals scored, a list of committed fouls, a list of corners taken. However, you must admit – the domain of a football match is ideally described by a series of events that have occurred over time.

Event Sourcing state management

When using Event Sourcing to manage the state of the Game object, we can accurately reproduce the entire game. We have information about what events have affected the current state of the object. The above picture shows that each event is reflected in a specific class. That’s the magic of Event Sourcing.

One of the main Event Sourcing advantages that is mentioned in most articles is that you do not lose any information. In traditional models, every update erases the previous state. The previous state is lost. You can say that there are logs, backups, and libraries like Envers but they do not provide you with explicit information about the reason for the changes. They show you what data has changed, but not why. In an event sourced approach, you model your events after business events in your domain, therefore it not only shows data changes but the reason for the change.

The next advantage is that saving your domain aggregates as a series of events which greatly simplifies your persistence model. You no longer have to design tables and the relations between them. You are no longer constrained by what your ORM can and cannot map. While working with extremely advanced solutions like Hibernate we found cases where we had to resign from some design concepts in our domain because it was hard or impossible to map to the database.

There are more and more solutions supporting the creation of applications that use Event Sourcing (EventStore, Streamstone, Marten, Axon, Eventuate). In our examples, we use our own implementation of in-memory event store (Java example, C# example) that is derived from Greg Young’s example. This is not a production-ready implementation. For a production grade solution, you should apply more complex solution, like EventStore or Axon.

In Which Systems Is it Worth Using Event Sourcing?

Your system has many behaviors that are not an ordinary CRUD.
Recreating historical states of objects is very important.
Business users see advantages of having full history for statistics, machine learning, or other purposes.
Your domain is best described by events (for example, an application to track an assistance vehicle's activities).

Should I Use CQRS/ES Frameworks?

If you are not experienced with CQRS/ES, you should not start with any framework. Start with your core domain stuff and implement some business functionality. When your business stuff starts to work, focus on the technical stuff. Evaluate the available choices like Event Store or Axon before going to implement your own event store or command bus. There many things to consider and many traps along the way (concurrency, error handling, versioning, schema migration).

Summary

There two camps: one says you should always use CQRS/ES; the other says you should use them only for parts of your solution and only when you need highly concurrent systems with high performance/availability/scalability.

You should always evaluate your choices in the context of your requirements.

Even the simplest form of CQRS gives you good results without introducing much complexity. For example, using views for searches instead of using a domain model simplifies things a lot. In our systems we also found lot of places where adding specialized read model tables and updating them synchronously gave us very good results (like getting rid of a 20+ tables join view definition with 4 unions and replacing them with one table). Using a dedicated search engine like ElasticSearch is also a safe bet, as long as eventual consistency is allowed.

CQRS may result in a very complex technical solution if you choose to use different storage engines, event buses, and other technical components. Only some complex scenarios and scalability requirements justify this complexity (if you run on a Netflix scale). At the same time you can also apply CQRS using simple technical solutions and benefit from this pattern – you don’t need Kafka to do CQRS.

Here, at Software House Altkom Software and Consulting, we are fascinated with new technologies and frameworks which allow us to improve our solutions. However, we are aware that every change costs and is an investment that should gives our customers business benefits. We don’t use Hype Driven Development or Resume Driven Development in everyday work. We solve real-life problems using well-crafted pragmatic solutions.

We prepared two version of demo for this blog post, one is for Java developers and second is for .NET developers. Links below:

.NET CQRS Intro: https://github.com/asc-lab/dotnet-cqrs-intro
Java CQRS Intro: https://github.com/asc-lab/java-cqrs-intro

At the end, for the most tenacious, a brief summary of the pros and cons of CQRS and Event Sourcing.

CQRS Pros and Cons

Pros:

Better performance and scalability of your system.
Better concurrent access handling.
Better team scalability.
Less complex domain model and simple query model.

Cons:

Read and write models must be kept in sync.
Maintenance and administration costs if you choose two different engines for the read and write sides.
Eventual consistency is not always allowed.

Event Sourcing Pros and Cons