Develop a Scalable Event Listener for Blockchain

Developing a Scalable Event Listener for Hyperledger Fabric

Using an event listener for replicating data stored in Hyperledger Fabric ledgers is an emerging pattern in todays’ enterprise solutions based on Hyperledger Fabric oriented distributed ledger technology-based platform. This article will provide a comprehensive guide for building an event listener using Hyperledger fabric-sdk-go. All the examples have assumed the usage of Hyperledger Fabric v2.2.x platform.

Why We Need Event Listeners

For many blockchain-based applications, it has been observed that there is a requirement of storing all or some of the assets into an off-chain data storage. There could be several motivations for such requirements. Some of the frequently encountered cases are

• To provide a faster search response for UI/other application-generated queries for assets stored in the ledger. In this case, an off-chain database could be an in-memory database or search optimized database
• To run analytics on the asset data stored in blockchain. Running analytical queries on blockchain peers is not a scalable and preferred option always. Copying assets into analytics optimized database is always preferred.

To implement such replication of assets in another place it best implemented using an event listener component. It is easier to comprehend and manage such event driver architecture.

Apart from copying data from a ledger to different storage, there could be a requirement of triggering a workflow inside a participating organization when an asset is created or a state of the asset is changed. In that situation, also listening to blockchain ledger state changes is the only key to initiate such a workflow.

Hence the use of a fabric event listener component is becoming a common design pattern that should be considered in all blockchain-based enterprise echo systems. It plays an important role in achieving key functionalities of replicating records in off-chain databases as well helps in proving scalability of the overall ecosystem.

Different Categories Hyperledger Fabric Blockchain Events

In Hyperledger Fabric, when the blockchain network commits records into the ledger channel, then 3 categories of events are emitted by the Hyperledger Fabric network to the hooked event listeners. They are block events, transaction events for those transactions contained in the block, and custom chaincode events.

• Custom chaincode events: If a smart contact uses stub.SetEvent(“<event name>”,”<event payload>”), during endorsement cycle, then after committing the blocks in the peer, these custom events are emitted by the peers. The smart contracts may choose to put the updated state of the committing asset as an event payload, and the event listener may capture these events and pull out the event payload for the latest state of the asset and then put it in the off-chain store. However, there is one drawback of this approach. If the event listener component fails during its operation and becomes unavailable, then during its unavailability, any event emitted by the peer is not replayed back. Hence the off-chain store will be out of sync with the ledger.
• Transaction event: Whenever a transaction is committed, then peers emit a transaction event. Any listener who has registered with the specific transaction of the transaction committed is notified. Then the event listener may read the transaction details using the ledger.Client. QueryTransaction method. From the transaction, envelop bytes required asset state could be retrieved from the write set. This approach too has the following drawbacks. First of all, the transaction ID is known to the client application, hence these event listeners could only be embedded inside the transaction client. Like the custom events, there is a chance of losing the data in case of client failure.
• Block event: When a block is committed into peers, peers emit block events. Any listener who has registered for the block events shall receive the block details. The event listener may use ledger.Client.QueryBlock function to retrieve the entire block details. The fabric block contains list of transactions. Each transaction entry contains chaincode invocation specification, all endorsement responses, world state key name and versions that were read ( read-set) during the smart contract execution, world state key name and value those are written into the world-state as a result of transaction commit ( write-set) and all the necessary signatures. Transaction entries also store chain code event and event payload.
  
  Please note that blocks do not store any information about the private data collection except for the private datahash. Since every shared data about a transaction is available in a block, it provides a superset of functions if an event listener reads and decodes the block events. It can read individual transaction details, custom event details, as well as all asset details that are being updated because of a transaction. Moreover, block listeners could be written in a fail-safe way so that even in case of listener failure, it is possible to recover from failure ensuring full synchronization of ledger assets with off-chain data.

Design of a Scalable Event Listener for Hyperledger Fabric

A scalable event listener should design should aim to resolve the following challenges

1. Should be able to read all the events and replicate the asset state in off-chain database
2. Should avoid any data loss
3. Should be able to scale with the transaction volume increases

Figure 1 shows a scalable design for event listener

Figure 1 - Design for Event Listener

• Light Weight Event Listener: This component receives the block events from the blockchain peer. This component is registered into a set of peers during its startup. Upon receiving block events this component invokes ledger.Client.QueryBlock to get the block payload using the block number received along with the block event notification. After the block is decoded, it may apply filter criteria like specific custom event, chaincode invocation, etc. to select specific transactions from the set of transactions written in the block. From the write set of the filtered transaction, this component decodes the write sets and places them into a message queue. Once block messages are decoded and placed in the queue, the latest block number that is processed is written into a file.
  
  In case of failure of the event listener, the component during its recovery. may read the last block read and the latest event block number. Then it may run a loop to read the missed block number to catch-up with missed transaction during its outage. Using this recovery mechanism this component will never miss a block event. There could be multiple instances of these event listeners dedicated to read blocks from different channels of Hyperledger Fabric. Depending on the workload and throughput required, more granular filter logic could be implemented during the block decoding process to route assets to different message queues dedicates to different off-chain storage or some other dedicated processing logic.
• Message queues: These message queues bridge the speed gap between the Lightweight Event Listener and the DB writer. These message queues must have a delivery guaranty and should support consumer group reads. Also, these queues act as temporary storage of records in case of its’ consumers failed to consume the data. For many analytic platforms, these queues may act a connector to feed data from blockchain.
• DB writer: This component reads the incoming messages from the message queues and performs the following activities
  1. Transform the asset to target data format (e.g. JSON to XML, RDBMS, etc.)
  2. Runs necessary business logic before inserting records in the database or may add additional information to the incoming asset
  3. Optionally may perform a duplicate check whenever required
  4. Submits the records in the off-chain DB using appropriate driver

This DB writer component may not be required if the target off-chain database platform can consume assets directly from message queues. This DB writer component is tightly coupled with the technology of the off-chain database. Depending on the system load and throughput required multiple instances of DB writer component could be utilized using a consumer group.

Salient Feature of This Design

The design discussed above resolves the following challenges

1. Resolves scalability challenges: If the number of blocks generated from the blockchain increases, the load of data replication could be shared using more instances of Lightweight Event Listeners. Each instance may be dedicated to read different channels; different transaction based on smart contract id, etc.
2. Handles missing data challenges: Since the design of the Lightweight Event Listener implements recovery mechanisms for missing blocks, hence target system will achieve zero data loss even if the event listener system fails. Once the event listener places the asset records in the persistent queues, data loss will never happen.
3. Addresses reusability challenges: This design is reusable for any Hyperledger Fabric projects and the listener code (Shared in the code repository) is highly reusable. The DB writer component is loosely coupled with the event listener. Even any suitable technology to could be utilized to develop the DB writer component to replicate the assets into a different database platform.

Step 1 - Initialize the listener

Figure 2: Initialize the listener

Step 2 - Run the block consumption loop 

Figure 3: Run the block consumption loop

Step 3 - Get Block Details

Figure 4: Get Block Details

Step 4 - Simplified block structure 

Figure 5: Simplified block structure

Step 5 - Decode block 

Figure 6 - Decode block

Conclusion

This article provided a comprehensive guide for building a scalable event listener for replicating data for Hyperledger Fabric into replicating records in an off-chain database in achieving a high level of scalability.