Exploring Hazelcast With Spring Boot

For the use cases I am going to describe here, I have two services: courses-service and reviews-service:

1. Courses-service provides CRUD operations for dealing with courses and instructors.
2. Reviews-service is another CRUD operations provider for dealing with reviews for courses that are completely agnostic of courses from courses-service. 

Both apps are written in Kotlin using Spring Boot and other libraries. Having these two services, we are going to discuss distributed caching with Hazelcast and Spring Boot and see how we can use user code-deployment to invoke some code execution via Hazelcast on a service. 

Spoiler alert: The examples/use cases presented here are designed purely for the sake of demonstrating integration with some of Hazelcast’s capabilities. The discussed problems here can be solved in various ways and maybe even in better ways, so don’t spend too much on thinking, “why?” So, without further ado, let’s dive into code.

Note: here is the source code in case you want to follow along.

Simple Distributed Caching

We’ll focus on courses-service for now. Having this entity:

@Entity
@Table(name = "courses")
class Course(
    var name: String,
    @Column(name = "programming_language")
    var programmingLanguage: String,
    @Column(name = "programming_language_description", length = 3000, nullable = true)
    var programmingLanguageDescription: String? = null,
    @Enumerated(EnumType.STRING)
    var category: Category,

    @ManyToOne(fetch = FetchType.LAZY)
    @JoinColumn(name = "instructor_id")
    var instructor: Instructor? = null
) : AbstractEntity() {

    override fun toString(): String {
        return "Course(id=$id, name='$name', category=$category)"
    }
}

And this method in CourseServiceImpl:

@Transactional
override fun save(course: Course): Course {
    return courseRepository.save(course)
}

I want to enhance every course that is saved with a programming language description for the programming language that has been sent by the user. For this, I created a Wikipedia API client that will make the following request every time a new course is added:

GET https://en.wikipedia.org/api/rest_v1/page/summary/java_(programming_language)

So, my method looks like this now:

@Transactional
override fun save(course: Course): Course {
    enhanceWithProgrammingLanguageDescription(course)
    return courseRepository.save(course)
}

private fun enhanceWithProgrammingLanguageDescription(course: Course) {
    wikipediaApiClient.fetchSummaryFor("${course.programmingLanguage}_(programming_language)")?.let { course.programmingLanguageDescription = it.summary }
}

Now here comes our use case, we want to cache the Wikipedia response so we don’t call every single time. Our courses will be mostly oriented to a set of popular programming languages like Java, Kotlin, C#, and popular programming languages. We also don’t want to decrease our save()’s performance by querying every time for mostly the same language. Also, this can act as a guard in case the API server is down.

Time to introduce Hazelcast!

Hazelcast is a distributed computation and storage platform for consistently low-latency querying, aggregation and stateful computation against event streams and traditional data sources. It allows you to quickly build resource-efficient, real-time applications. You can deploy it at any scale from small edge devices to a large cluster of cloud instances.

You can read about lots of places where Hazelcast is the appropriate solution and all the advantages on their homepage.

When it comes to integrating a Spring Boot app with Hazelcast (embedded), it is straightforward. There are a few ways of configuring Hazelcast, via XML, YAML, or the programmatic way. Also, there is a nice integration with Spring Boot’s cache support via @EnableCaching and @Cacheable annotations. I picked the programmatic way of configuring Hazelcast and the manual way of using it—a bit more control and less magic.

Here are the dependencies:

implementation("com.hazelcast:hazelcast:5.2.1")
implementation("com.hazelcast:hazelcast-spring:5.2.1")

And here is the configuration that we are going to add to courses-service:

@Configuration
class HazelcastConfiguration {

    companion object {
        const val WIKIPEDIA_SUMMARIES = "WIKIPEDIA_SUMMARIES"
    }

    @Bean
    fun managedContext(): SpringManagedContext {
        return SpringManagedContext()
    }


    @Bean
    fun hazelcastConfig(managedContext: SpringManagedContext): Config {
        val config = Config()
        config.managedContext = managedContext
        config.networkConfig.isPortAutoIncrement = true
        config.networkConfig.join.multicastConfig.isEnabled = true
        config.networkConfig.join.multicastConfig.multicastPort = 5777
        config.userCodeDeploymentConfig.isEnabled = true
        config.configureWikipediaSummaries()
        return config
    }



    private fun Config.configureWikipediaSummaries() {
        val wikipediaSummaries = MapConfig()
        wikipediaSummaries.name = WIKIPEDIA_SUMMARIES
        wikipediaSummaries.isStatisticsEnabled = true
        wikipediaSummaries.backupCount = 1
        wikipediaSummaries.evictionConfig.evictionPolicy = EvictionPolicy.LRU
        wikipediaSummaries.evictionConfig.size = 10000
        wikipediaSummaries.evictionConfig.maxSizePolicy = MaxSizePolicy.PER_NODE

        addMapConfig(wikipediaSummaries)
    }
}

So we declare a managedContext() bean, which is a container-managed context initialized with a Spring context implementation that is going to work along with the @SpringAware annotation to allow us to initialize/inject fields in deserialized instances. We’ll take a look at why we need this later when we discuss user code deployment.

Then, we declare a hazelcastConfig() bean, which represents the brains of the whole integration. We set the managedContext, enable user code deployment, and set the networkConfig’s join option to “multicast.” Basically, the NetworkConfig is responsible for defining how a member will interact with other members or clients—there are multiple parameters available for configuration like port, isPortAutoIncrement, sslConfig, restApiConfig, joinConfig, and others. We configured the isPortAutoIncrement to true to allow hazelcastInstance to auto-increment the port if the picked one is already in use until it runs out of free ports. Also, we configured the JoinConfig, which contains multiple member/client join configurations like Eureka, Kubernetes, and others. We enable MulticastConfig, which allows Hazelcast members to find each other without the need to know concrete addresses via multicasting to everyone listening. Also, I encountered some issues with the port used by Multicast, so I set it to a hard-coded one to avoid the address already in use.

Then we configure a Map config that is going to act as our distributed cache. MapConfig contains the configuration of an IMap—concurrent, distributed, observable, and queryable map:

• name: the name of IMap, WIKIPEDIA_SUMMARIES.
• isStatisticsEnabled: this is to enable IMap’s statistics like the total number of hits and others.
• backupCount: number of synchronous backups, where “0” means no backup.
• evictionConfig.evictionPolicy: can be LRU (Least Recently Used), LFU (Least Frequently Used), NONE, and RANDOM. 
• evictionConfig.size: the size used by MaxSizePolicy.
• evictionConfig.maxSizePolicy: PER_NODE (policy based on the maximum number of entries per the Hazelcast instance). 

Having this configuration, all that is left is to adjust our enhanceWithProgrammingLanguageDescription method to use the above configured IMap to cache the fetched Wikipedia summaries:

private fun enhanceWithProgrammingLanguageDescription(course: Course) {
    val summaries = hazelcastInstance.getMap<String, WikipediaApiClientImpl.WikipediaSummary>(WIKIPEDIA_SUMMARIES)
    log.debug("Fetched hazelcast cache [$WIKIPEDIA_SUMMARIES] = [${summaries}(${summaries.size})]  ")
    summaries.getOrElse(course.programmingLanguage) {
        wikipediaApiClient.fetchSummaryFor("${course.programmingLanguage}_(programming_language)")?.let {
            log.debug("No cache value found, using wikipedia's response $it to update $course programming language description")
            summaries[course.programmingLanguage] = it
            it
        }
    }?.let { course.programmingLanguageDescription = it.summary }
}

Basically, we are using the autowired Hazelcast instance to retrieve our configured IMap. Each instance is a member and/or client in a Hazelcast cluster. When you want to use Hazelcast’s distributed data structures, you must first create an instance. In our case, we simply autowire it as it will be created by the previously defined config. After getting a hold of the distributed Map, it is a matter of some simple checks. If we have a summary for the programming language key in our map, then we use that one. If not, we fetch it from Wikipedia API, add it to the map, and use it.

Now, if we are to start our app, we’ll first see the huge Hazelcast banner and the following lines, meaning that Hazelcast has started:

INFO 30844 --- [           main] com.hazelcast.core.LifecycleService      : [ip]:5701 [dev] [5.2.1] [ip]:5701 is STARTING
INFO 30844 --- [           main] c.h.internal.cluster.ClusterService      : [ip]:5701 [dev] [5.2.1] 
Members {size:1, ver:1} [
	Member [ip]:5701 - e2a90d3e-b112-4e78-aa42-58a959d9273d this
]
INFO 30844 --- [           main] com.hazelcast.core.LifecycleService      : [ip]:5701 [dev] [5.2.1] [ip]:5701 is STARTED

If we execute the following HTTP request:

POST http://localhost:8081/api/v1/courses
Content-Type: application/json

{
  "name": "C++ Development",
  "category": "TUTORIAL",
  "programmingLanguage" : "C++",
  "instructor": {
    "name": "Bjarne Stroustrup"
  }
}

We’ll see in the logs:

DEBUG 30844 --- [nio-8080-exec-1] i.e.c.s.i.CourseServiceImpl$Companion    : Fetched hazelcast cache [WIKIPEDIA_SUMMARIES] = [IMap{name='WIKIPEDIA_SUMMARIES'}(0)]  
INFO 30844 --- [nio-8080-exec-1] e.c.s.i.WikipediaApiClientImpl$Companion : Request GET:https://en.wikipedia.org/api/rest_v1/page/summary/C++_(programming_language)
INFO 30844 --- [nio-8080-exec-1] e.c.s.i.WikipediaApiClientImpl$Companion : Received response from Wikipedia...
DEBUG 30844 --- [nio-8080-exec-1] i.e.c.s.i.CourseServiceImpl$Companion    : No cache value found, using wikipedia...

That we retrieved the previously configured IMap for Wikipedia summaries, but its size is “0;” therefore, the update took place using the Wikipedia’s API. Now, if we are to execute the same request again, we’ll notice a different behavior:

DEBUG 30844 --- [nio-8080-exec-3] i.e.c.s.i.CourseServiceImpl$Companion    : Fetched hazelcast cache [WIKIPEDIA_SUMMARIES] = [IMap{name='WIKIPEDIA_SUMMARIES'}(1)]  

Now the IMap has a size of “1” since it was populated by our previous request, so no request to Wikipedia’s API can be observed. The beauty and simplicity of Hazelcast’s integration comes when we start another instance of our app on a different port using -Dserver.port=8081, and we witness the distributed cache in action.

INFO 8172 --- [           main] com.hazelcast.core.LifecycleService      : [ip]:5702 [dev] [5.2.1] [ip]:5702 is STARTING
INFO 8172 --- [           main] c.h.i.cluster.impl.MulticastJoiner       : [ip]:5702 [dev] [5.2.1] Trying to join to discovered node: [ip]:5701
INFO 8172 --- [.IO.thread-in-0] c.h.i.server.tcp.TcpServerConnection     : [ip]:5702 [dev] [5.2.1] Initialized new cluster connection between /ip:55309 and /ip:5701
INFO 8172 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [ip]:5702 [dev] [5.2.1] 
Members {size:2, ver:2} [
	Member [ip]:5701 - 69d6721d-179b-4dc8-8163-e3cb00e703eb
	Member [ip]:5702 - 9b689155-d4c3-4169-ae53-ff5d687f7ad2 this
]
INFO 8172 --- [           main] com.hazelcast.core.LifecycleService      : [ip]:5702 [dev] [5.2.1] [ip]:5702 is STARTED

We see that MulticastJoiner discovered an already running Hazelcast node on port 5701, running together with our first courses-service instance on port 8080. A new cluster connection is made, and we see in the “Members” list both Hazelcast nodes on ports 5701 and 5702. Now, if we are to make a new HTTP request to create a course on the 8081 instance, we’ll see the following:

DEBUG 8172 --- [nio-8081-exec-4] i.e.c.s.i.CourseServiceImpl$Companion    : Fetched hazelcast cache [WIKIPEDIA_SUMMARIES] = [IMap{name='WIKIPEDIA_SUMMARIES'}(1)]  

Distributed Locks

Another useful feature that comes with Hazelcast is the API for distributed locks. Suppose our enhanceWithProgrammingLanguageDescription method is a slow intensive operation dealing with cache and other resources and we wouldn’t want other threads on the same instance or even other requests on a different instance to interfere or alter something until the operation is complete. So, here comes FencedLock into play—a linearizable, distributed, and reentrant implementation of the Lock. It is consistent and partition tolerant in the sense that if a network partition occurs, it will stay available on, at most, one side of the partition. Mostly, it offers the same API as the Lock interface. So, with this in mind, let’s try and guard our so-called “critical section.”

private fun enhanceWithProgrammingLanguageDescription(course: Course) {
    val lock = hazelcastInstance.cpSubsystem.getLock(SUMMARIES_LOCK)
    if (!lock.tryLock()) throw LockAcquisitionException(SUMMARIES_LOCK, "enhanceWithProgrammingLanguageDescription")
    Thread.sleep(2000)
    val summaries = hazelcastInstance.getMap<String, WikipediaApiClientImpl.WikipediaSummary>(WIKIPEDIA_SUMMARIES)
    log.debug("Fetched hazelcast cache [$WIKIPEDIA_SUMMARIES] = [${summaries}(${summaries.size})]  ")
    summaries.getOrElse(course.programmingLanguage) {
        wikipediaApiClient.fetchSummaryFor("${course.programmingLanguage}_(programming_language)")?.let {
            log.debug("No cache value found, using wikipedia's response $it to update $course programming language description")
            summaries[course.programmingLanguage] = it
            it
        }
    }?.let { course.programmingLanguageDescription = it.summary }
    lock.unlock()
}

As you can see, the implementation is quite simple. We obtain the lock via the cpSubsystem’s, getLock(), and then we try acquiring the lock with tryLock(). The locking will succeed if the acquired lock is available or already held by the current thread, and it will immediately return true. Otherwise, it will return false, and a LockAcquisitionException will be thrown. Next, we simulate some intensive work by sleeping for two seconds with Thread.sleep(2000), and in the end, we release the acquired lock with unlock(). If we run a single instance of our app on port 8080 and try two subsequent requests, one will pass, and the other one will fail with:

ERROR 28956 --- [nio-8081-exec-6] e.c.w.r.e.RESTExceptionHandler$Companion : Exception while handling request [summaries-lock] could not be acquired for [enhanceWithProgrammingLanguageDescription] operation. Please try again.
inc.evil.coursecatalog.common.exceptions.LockAcquisitionException: [summaries-lock] could not be acquired for [enhanceWithProgrammingLanguageDescription] operation. Please try again

The same goes if we are to make one request to an 8080 instance of our app and the next one in the two seconds timeframe to the 8081 instance; the first request will succeed while the second one will fail.

User Code Deployment

Now, let’s switch our attention to reviews-service and remember—this service is totally unaware of courses; it is just a way to add reviews for some course_id. With this in mind, we have this entity:

@Table("reviews")
data class Review(
    @Id
    var id: Int? = null,
    var text: String,
    var author: String,
    @Column("created_at")
    @CreatedDate
    var createdAt: LocalDateTime? = null,
    @LastModifiedDate
    @Column("last_modified_at")
    var lastModifiedAt: LocalDateTime? = null,
    @Column("course_id")
    var courseId: Int? = null
)

And we have this method in ReviewServiceImpl :

override suspend fun save(review: Review): Review {
    return reviewRepository.save(review).awaitFirst()
}

So, our new silly feature request would be to somehow check for the existence of the course that the review has been written for. How can we do that? The most obvious choice would be to invoke a REST endpoint on courses-service to check if we have a course for the review’s course_id, but that is not what this article is about. We have Hazelcast, right? We are going to deploy some user code from reviews-service that courses-service is aware of and can execute it via Hazelcast’s user code deployment.

To do that, we need to create some kind of API or gateway module that we are going to publish as an artifact, so courses-service can implement it, and reviews-service can depend on and use it to deploy the code. First things first, let’s design the new module as a courses-api module:

plugins {
    id("org.springframework.boot") version "2.7.3"
    id("io.spring.dependency-management") version "1.0.13.RELEASE"
    kotlin("jvm") version "1.6.21"
    kotlin("plugin.spring") version "1.6.21"
    kotlin("plugin.jpa") version "1.3.72"
    `maven-publish`
}

group = "inc.evil"
version = "0.0.1-SNAPSHOT"

repositories {
    mavenCentral()
}

publishing {
    publications {
        create<MavenPublication>("maven") {
            groupId = "inc.evil"
            artifactId = "courses-api"
            version = "1.1"

            from(components["java"])
        }
    }
}

dependencies {
    implementation("org.springframework.boot:spring-boot-starter-actuator")
    implementation("org.springframework.boot:spring-boot-starter-web")
    implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
    implementation("org.jetbrains.kotlin:kotlin-reflect")
    implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-rx2:1.6.4")
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4")
    implementation("org.apache.commons:commons-lang3:3.12.0")
    implementation("com.hazelcast:hazelcast:5.2.1")
    implementation("com.hazelcast:hazelcast-spring:5.2.1")

    testImplementation("org.junit.jupiter:junit-jupiter-api:5.8.1")
    testRuntimeOnly("org.junit.jupiter:junit-jupiter-engine:5.8.1")
}

tasks.getByName<Test>("test") {
    useJUnitPlatform()
}

Nothing fancy here, except the maven-publish plugin that we’ll use to publish the artifact to the local maven repository.

Here is the interface that courses-service will implement, and reviews-service will use:

interface CourseApiFacade {
    fun findById(id: Int): CourseApiResponse
}

data class InstructorApiResponse(
    val id: Int?,
    val name: String?,
    val summary: String?,
    val description: String?
)

data class CourseApiResponse(
    val id: Int?,
    val name: String,
    val category: String,
    val programmingLanguage: String,
    val programmingLanguageDescription: String?,
    val createdAt: String,
    val updatedAt: String,
    val instructor: InstructorApiResponse
)

Having this module properly configured, we can add it as a dependency in courses-service:

implementation(project(":courses-api"))

And implement the exposed interface like this:

@Component
class CourseApiFacadeImpl(val courseService: CourseService) : CourseApiFacade {

    override fun findById(id: Int): CourseApiResponse = courseService.findById(id).let {
        CourseApiResponse(
            id = it.id,
            name = it.name,
            category = it.category.toString(),
            programmingLanguage = it.programmingLanguage,
            programmingLanguageDescription = it.programmingLanguageDescription,
            createdAt = it.createdAt.toString(),
            updatedAt = it.updatedAt.toString(),
            instructor = InstructorApiResponse(it.instructor?.id, it.instructor?.name, it.instructor?.summary, it.instructor?.description)
        )
    }

}

Now back to reviews-service where all the magic will happen. First of all, we want to add the required dependencies for Hazelcast:

implementation("com.hazelcast:hazelcast:5.2.1")
implementation("com.hazelcast:hazelcast-spring:5.2.1")

Previously, I mentioned that we are going to use that interface from courses-api. We can run the publishMavenPublicationToMavenLocal gradle task on courses-api to get our artifact published, and then we can add the following dependency to reviews-service:

implementation("inc.evil:courses-api:1.1")

Now, is time to set up a Callable implementation on reviews-service that will be responsible for the code execution on courses-service, so, here it is:

@SpringAware
class GetCourseByIdCallable(val id: Int) : Callable<CourseApiResponse?>, Serializable {

    @Autowired
    @Transient
    private lateinit var courseApiFacade: CourseApiFacade

    override fun call(): CourseApiResponse = courseApiFacade.findById(id)
}

Here we used the @SpringAware annotation to mark this class as a bean in Spring-Hazelcast way via SpringManagedContext since this class is going to be deployed in the cluster. Other than that, we have our courseApiFacade autowired and used in the overridden method call() from Callable interface. I decided to write another class to ease the Callable submission to the Hazelcast’s IExecutorService to act as some kind of a facade:

@Component
class HazelcastGateway(private val hazelcastInstance: HazelcastInstance) {

    companion object {
        private const val EXECUTOR_SERVICE_NAME = "EXECUTOR_SERVICE"
    }

    fun <R> execute(executionRequest: Callable<R>): R {
        val ex = hazelcastInstance.getExecutorService(EXECUTOR_SERVICE_NAME)
        return ex.submit(executionRequest).get(15000L, TimeUnit.MILLISECONDS)
    }
}

We have a single method execution that is responsible to submit the passe—in Callable to the retrieved IExecutorService via the autowired hazelcastInstance. Now, this IExectuorService is a distributed implementation of ExecutorService that enables the running of Runnables/Callables on the Hazelcast cluster and has some additional methods that allows running the code on a particular member or multiple members. For example, we’ve used the submit, but there is also submitToMember and submitToAllMembers. For Runnable, there are the equivalents that start with execute***.

Now, let’s use our newly defined HazelcastGateway in the save method from ReviewServiceImpl.

override suspend fun save(review: Review): Review {
    runCatching {
        hazelcastGateway.execute(GetCourseByIdCallable(review.courseId!!)).also { log.info("Call to hazelcast ended with $it") }
    }.getOrNull() ?: throw NotFoundException(CourseApiResponse::class, "course_id", review.courseId.toString())
    return reviewRepository.save(review).awaitFirst()
}

The logic is as follows: before saving, we try to find the course by course_id from review by running GetCourseByIdCallable in our Hazelcast cluster. If we have an exception (CourseApiFacadeImpl will throw a NotFoundException if the requested course was not found), we swallow it and throw a reviews-service NotFoundException stating that the course could’ve not been retrieved. If a course was returned by our Callable, we proceed to save it—that’s it.

All that is left is to configure Hazelcast, and I left this one last since we needed some of these classes to configure it. 

@Configuration
class HazelcastConfiguration {


    @Bean
    fun managedContext(): SpringManagedContext {
        return SpringManagedContext()
    }

    @Bean
    fun hazelcastClientConfig(managedContext: SpringManagedContext): ClientConfig {
        val config = ClientConfig()
        config.managedContext = managedContext
        config.connectionStrategyConfig.isAsyncStart = true
        config.userCodeDeploymentConfig.isEnabled = true
        config.classLoader = HazelcastConfiguration::class.java.classLoader
                config.userCodeDeploymentConfig.addClass(GetCourseByIdCallable::class.java)

        return config
    }

}

Regarding the managedContext bean, there’s nothing different from our previous member’s config. Now the hazelcastConfig bean is different. First of all, it is a ClientConfig now, meaning that reviews-service’s embedded Hazelcast will be a client to the courses-service Hazelcast member. We set the managedContext, the connectionStrategyConfig to be async (the client won’t wait for a connection to cluster, it will throw exceptions until connected and ready), and we will enable the user code deployment. Then we add GetCourseByIdCallable to be sent to the cluster.

Now, having courses-service running and if we are to start reviews-service, we’ll see the following log:

INFO 34472 --- [nt_1.internal-1] com.hazelcast.core.LifecycleService      : hz.client_1 [dev] [5.2.1] HazelcastClient 5.2.1 (20221114 - 531032a) is CLIENT_CONNECTED
INFO 34472 --- [nt_1.internal-1] c.h.c.i.c.ClientConnectionManager        : hz.client_1 [dev] [5.2.1] Authenticated with server [ip]:5701:69d6721d-179b-4dc8-8163-e3cb00e703eb, server version: 5.2.1, local address: /127.0.0.1:57050

The client is authenticated with courses-service’s Hazelcast member, where “69d6721d-179b-4dc8-8163-e3cb00e703eb” is the ID of the connected to server. Now, let’s try a request to add a review for an existing course (reviews-service is using GraphQL).

GRAPHQL http://localhost:8082/graphql
Content-Type: application/graphql

mutation { createReview(request: {text: "Amazing, loved it!" courseId: 2 author: "Mike Scott"}) {
    id
    text
    author
    courseId
    createdAt
    lastModifiedAt
}
}

In the logs, we’ll notice:

INFO 34472 --- [actor-tcp-nio-1] i.e.r.s.i.ReviewServiceImpl$Companion    : Call to hazelcast ended with CourseApiResponse(id=2, name=C++ Development, category=TUTORIAL, programmingLanguage=C++ ...)

And in the courses-service logs, we’ll notice the code execution:

DEBUG 12608 --- [ached.thread-24] i.e.c.c.aop.LoggingAspect$Companion      : before :: execution(public inc.evil.coursecatalog.model.Course inc.evil.coursecatalog.service.impl.CourseServiceImpl.findById(int))

Meaning that the request executed successfully. If we try the same request for a non-existent course, let’s say for ID 99, we’ll observe the NotFoundException in reviews-service: 

WARN 34472 --- [actor-tcp-nio-2] .w.g.e.GraphQLExceptionHandler$Companion : Exception while handling request: CourseApiResponse with course_id equal to [99] could not be found! 
inc.evil.reviews.common.exceptions.NotFoundException: CourseApiResponse with course_id equal to [99] could not be found!

Conclusion

All-right folks, this is basically it. I hope you got a good feel of what Hazelcast is like. We took a look at how to design a simple distributed cache using Hazelcast and Spring Boot, made use of distributed locks to protect critical sections of code, and in the end, we’ve seen how Hazelcast’s user code deployment can be used to run some code in the cluster. In case you’ve missed it, all the source code can be found here.

Happy coding!