Joe has a habit, whenever he makes a transaction with his debit card, of writing it in his diary so he can track his transactions. Joe is not technology savvy and is not able to check account statements online.
At the month's end, his bank sends him his current balance, but he immediately notices a discrepancy between his current savings the bank shows and his savings as per his calculation based on the diary. He immediately calls the bank helpline and argues about the discrepancy. Then, the bank sends him an account statement with all transactions recorded.
When he is trying to match the transaction records with his diary, he understands that one transaction is not in the diary as that day he was so sick, he thought to write it next day but somehow forgot.
The Question Is, What Can We Extract From This Story?
If we look closely, we discover that the bank always stores all the events/transactions that happen on the account, like account creation, credit, debit, etc., and the current balance is nothing but the outcome of those transactions. What I meant to say is that account balance is not a fixed column in a database; rather, it is a derivative/outcome of the transactions or events that were applied to the account. We called it event sourcing.
Now, think of what we generally do in software. If we credit or debit any amount, we just add or subtract that amount from the current balance and update the account with the new balance, right? We have the current state but have lost the information how that state was achieved. Some systems use audit trails; still, it is not fully deterministic. So anytime anyone challenges the system saying that this is not the desired system state, we don’t have any solid proof other than pleading with them that the system can not be wrong. But if you maintain that history or cause of the state changes, like the bank, then you just give them the history and ask to check- a solid way to store proofs.
This is a very common story. Anyone of us has gone through the same thing and then looked at the account statement to clear our doubts.
Technically, What Is Event Sourcing?
Event sourcing is a technique by which we store all the changes of an application state as a sequence of events. We can rebuild the state anytime from those events, and can also query the events to construct the desired state.
The two key benefits are:
We store all the events in a sequence, which enables huge opportunities.
The state is the derivative of the events, so we don’t have to maintain state in the database; rather we can programmatically derive the state based on the event.
This opens a new direction where we don’t have to persist the state; rather, we can derive the state, and it brings many advantages
I will talk about five such advantages.
1. State Rebuild
As we store every event applied to an application object, we can create a blank/initial application object and apply every event in the same sequence it was applied. Doing this will bring the same state, so anywhere- any point of time- we can rebuild a state from events. Systems must have a mechanism to apply events, then you can rebuild a state if the state is blown up for some reason. One may argue that if your application state derives from millions of events applied on it, computing all events may take time, and storing all events takes a big storage area. But the fact, is nowadays, memory is really cheap. Also, we can have TB of memory space, so computation is also faster. Alternatively, we can store a snapshot, i.e. a milestone, of the state and apply events and rebuild the state from the latest snapshot.
2. Temporal Query
Event sourcing is perfect for auditors. Business analysis teams always want to see the past state so they can compare the growth or loss or any valuable statistical data, so they need the flexibility to query the system in all possible ways to collect statistical data. If the system has a feature to build the past state by passing parameters, the analyst team will be delighted. With the system which maintains all the states, they can easily rebuild/compute the state by the parameters provided by the analyst team. Say the analysts want to see the account details for the 10th of December, 2016. With event sourcing, we can fetch all events up until the 10th of December and apply them in sequence to build the state and return the result to the analysts- an easy job, isn’t it?
3. Comparing State
Sometimes, in a complex system, you need to know what the outcome would be and how much deviation it cause from the current state if events were applied in different ways. Say a bank savings account's interest rate is 8%, and previously it was 8.5%. Now, if the bank wants to know the actual amount the bank benefits due to the decrease of the interest, they will replay events of 8.5 percents in all accounts and compare the state with current state to know the actual benefits. It is not very easy to implement, but we can.
4. Debugging State
Suppose there is a bug in the production system and we need to figure out why the bug happens. With event sourcing, it is as easy as copying the account in the dev environment then changing the log level to debug, then applying the events one by one in the sequence and checking if the outcome is as predicted or not. If not, then find the event and check how it applies to change the application state to find the defect.
5. Future Prediction
In some business domains, it is an important task to analyze what the outcome will be if we make a certain business decision. If the outcome is successful, they will make the decision, but to the naked eye, it is impossible to predict the application state as different services are interlinked with each other, and based on one event, they can change. Dependent services are subscribed to certain events; when that event occurs, they take action on the basis of the event value. Say a bank’s stock share value is 8 INR, but the bank analysis team predicts that within one month, it will be increased to 12 INR, and they have 30K public stocks. The analysis team wants to know what the effects of the application state will be if the stock value is 12 INR, so they will run some ad-hoc future events on top of the current state, based on two criteria: taking per stock as 12 INR, and taking per stock as 8 INR.
Then, they can compare two application states to find out the effects of this stock value increase for each interlinked service.
Some systems are inherently event sourced, like version control (Git), banking applications, order tracking applications, etc. But we can implement the same in general systems, too. Using event sourcing, you can easily go back and forth in your application state by replaying events and state cloning into any environment; it's just a matter of time, but the irony is, this pattern not used broadly in industry.