A New Expiring Item Service For Microservice Environment: Tickler
In this article, learn about Tickler, a new expiring item service for a Microservice environment, and when you may need to use it.
Join the DZone community and get the full member experience.Join For Free
First, let's define the need for Tickler:
I am, as a Tickler client application, running in distributed microservice environment and I need an expiring distributed data structure in which all of the items in this structure may have different time-to-live values.
These JSON-based expiring items can be called reminders (tickles), which are composed of a callback URL of the client or callback Kafka topic which is consumed by the client, JSON payload for the callback, and a TTL value in seconds.
The main components of a Tickler client and Tickler main application nodes are stated on the image below:
The basic flow of the diagram:
- A client posts a tickle via a rest call or publishes a tickle to a Kafka topic. The expiry requested tickle contains a rest callback url or a callback Kafka topic, which will be triggered with the payload after the expiration. A tickle contains a time-to-live value (in seconds) beside the callbak info. Payload message is the data which will be sent back to the tickle client after the expiration.
- If it is Redis Background Enabled, a tickler instance that receives the tickle request sets two key-value pairs onto REDIS. One of them is for receiving the expired key and the other is receiving the expired value. The two key-value pairs must be set because REDIS gives only the key of the record when it expires, not both key and value. If it is Couchbase Background Enabled, a tickler instance that receives the tickle request inserts the tickle into a couchbase bucket with the requested TTL value. An eventing process, which is the core functionality of Couchbase cluster, sets a Couchbase timer after this insertion. The timer function runs when TTL expires and calls an endpoint of a Tickler instance, which will distribute the expired tickle to be processed among tickler instances.
- One of the tickler instances is stated as the leader for distributing the expired tickle if REDIS background is enabled. This is because REDIS pushes the key expiration events to all subscribed instances. If REDIS background is enabled, the expired tickle key is received by all tickler instances, but only the leader initiates the distribution of the expired tickle among the Tickler instances (Kafka pub-sub). The leader gets the tickle value from REDIS after receiving the expired tickle key. On the other hand, if Couchbase background is enabled, there is no need for leader election because Couchbase timer function makes an only one rest call to tickler instances.
- After having the expired tickle value, the tickler instance (it is the leader if REDIS background is enabled or any of tickler instances if Couchbase background is enabled) sends it to a specific Kafka topic (tickle.process), which is consumed by all the Tickler instances. This is done for spreading the tickle processing load on all tickler instances. All the tickler instances are bound to this internal topic with the same group id and the load is distributed among all tickler instances.
- Any tickler instance receives the tickle value from internal Kafka topic (tickle.process). Now, it has all the values to finish the process.
- It is time to call the client's callback url with the corresponding payload or send it to a callback Kafka topic, which is provided by the tickle client. The client is informed with the payload data of the tickle after the expiration.
If you want to contribute, please do not hesitate.
Opinions expressed by DZone contributors are their own.