DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
View Events Video Library
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Integrating PostgreSQL Databases with ANF: Join this workshop to learn how to create a PostgreSQL server using Instaclustr’s managed service

Mobile Database Essentials: Assess data needs, storage requirements, and more when leveraging databases for cloud and edge applications.

Monitoring and Observability for LLMs: Datadog and Google Cloud discuss how to achieve optimal AI model performance.

Automated Testing: The latest on architecture, TDD, and the benefits of AI and low-code tools.

Related

  • 3 Best Tools to Implement Kubernetes Observability
  • Data Fabric: What Is It and Why Do You Need It?
  • Implement a Distributed Database to Your Java Application
  • Why My In-Memory Cluster Underperforms: Negating Network Impact

Trending

  • Build Quicker With Zipper: Building a Ping Pong Ranking App Using TypeScript Functions
  • Time Series Analysis: VAR-Model-As-A-Service Using Flask and MinIO
  • Best GitHub-Like Alternatives for Machine Learning Projects
  • LLMs for Bad Content Detection: Pros and Cons
  1. DZone
  2. Testing, Deployment, and Maintenance
  3. DevOps and CI/CD
  4. Syncing GridGain and a Database Using Oracle GoldenGate

Syncing GridGain and a Database Using Oracle GoldenGate

The basics behind Oracle GoldenGate's ability to sync GridGain's In-Memory Computing Cluster with a database like MySQL or OracleDB.

Denis Magda user avatar by
Denis Magda
CORE ·
Nov. 13, 16 · Tutorial
Like (4)
Save
Tweet
Share
3.94K Views

Join the DZone community and get the full member experience.

Join For Free

Almost any In-Memory Data Grid (IMDG) solution available can be used as-is without an underlying persistent storage layer. Based on my experience, there are different use cases and real production scenarios when the entire data set is fully located in an IMDG and it is not synced to disk at all.

However, in a variety of deployments, companies still prefer to keep data both in memory and on disk. They do this to ensure that the data will not be completely lost if the whole IMDG cluster goes down or needs to be restarted.

Figure 1 below depicts one of the most popular deployments of a cluster running using GridGain In-Memory Data Grid feature. An application or service communicates directly to the cluster and stores data there, executing distributed transactions and SQL queries, sending computations and streaming real-time data sets. But the application never updates a database or reads data from it directly. 

Figure 1.

Figure 1

It is the responsibility of the GridGain engine to make sure that the database is always up-to-date by syncing all the updates that happen in RAM to disk. It is GridGain's responsibility to preload a missing key's value to RAM from the database, if the key's value is missing. Thanks to the GridGain Cache Store mechanism, this functionality works out of the box, making integration with popular databases like Oracle, MySQL or PostgreSQL trivial.

Database Modification By Many Sources

The configuration shown above would be perfect if this type of deployment was applicable for all the business cases. However, how can you deal with situations where a database is being updated by both the GridGain cluster and also other applications that interact with the database directly, similar to the situation shown in Figure 2 below? How can you guarantee that stale data won't reside on the GridGain cluster for a long time and will be updated as soon as the database receives updates from the other applications?

Figure 2.

Figure 2

A straightforward workaround would be to connect App 2 directly to the GridGain cluster rather than to the database. However, this might be impossible in some organizations due to historical reasons. In other cases, applications might be migrated to an in-memory data grid solution step-by-step which means that some applications  might already work over GridGain while the remainder are directly integrated with a database.

Another feasible solution is to implement custom logic on the GridGain cluster side. That logic could check the database periodically, get all the updates that have happened recently, and store them in RAM on the cluster side. While this is a realistic solution, it requires a lot of engineering effort because to make sure that only deltas of the updates are retrieved from the database and that the deltas are applied in a transactional fashion to the GridGain cluster.

What other options are available? Is there any alternative that will allow building architectures similar to the one shown in Figure 2? They must guarantee that the GridGain cluster will be updated whenever the database is modified by another source. The answer is yes! Let's look at how Oracle GoldenGate can help us out.

Syncing the GridGain In-Memory Computing Cluster and Database

With Oracle GoldenGate, Figure 2 is transformed into Figure 3 below.

Figure 3

Figure 3

As Oracle says, "Oracle GoldenGate is a comprehensive software package for real-time data integration and replication in heterogeneous IT environments. The product set enables high availability solutions, real-time data integration, transactional change data capture, data replication, transformations, and verification between operational and analytical enterprise systems."

Undoubtedly, this Oracle's statement is true and GridGain provides the integration with this software package making sure that deployments like the one in Figure 3 can be easily brought to life.

The GridGain Enterprise Edition Oracle GoldenGate Integration provides a solution for real-time integration and replication of GoldenGate compatible databases and a GridGain cluster. When GoldenGate integration replication is configured, GridGain will automatically receive updates from the configured GoldenGate source database and will store the updates on the GridGain side like it is depicted in Figure 3 above.

Moreover, Oracle GoldenGate doesn't lock you to Oracle Database. You are free to use one of the databases that aresupported by this software package. Among the supported databases you can find MySQL, Microsoft SQL Server, IBM DB2 and more. 

I'm not going to cover technical details in this blog post elaborating on how to set up a GridGain cluster with Oracle GoldenGate. This is already perfectly done in this technical documentation section. So, go ahead and try it out! And you will see how the integration can greatly simplify your life.

Database cluster Data integration application Data grid

Published at DZone with permission of Denis Magda. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • 3 Best Tools to Implement Kubernetes Observability
  • Data Fabric: What Is It and Why Do You Need It?
  • Implement a Distributed Database to Your Java Application
  • Why My In-Memory Cluster Underperforms: Negating Network Impact

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: