Modern Application Design With In-Memory Data Fabrics
Modern Application Design With In-Memory Data Fabrics
In-memory grids like Apache Ignite have served as an essential, architectural component for transforming the way businesses use their data to do business.
Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
Portions of this article were taken from the book High-Performance In-Memory Computing With Apache Ignite. If it got you interested, check out the rest of the book for more helpful information.
Let’s take a quick look at an architecture of a traditional system. Traditional application architecture uses data stores which have synchronous read-write operations. This is useful for data consistency and data durability, but it is very easy to get bottleneck if there are a lot of transactions waiting in the queue. Consider the following traditional architecture.
High-Volume Transaction Processing
In-memory data grid adds an additional layer within an environment, which uses the Random-Access Memory (RAM) of the server to store most of all data required by the applications. In-memory data grid sits between the application servers and the data store. In-memory data grid uses a cache of frequently accessed data by the client in the active memory and then can access the persistence store whenever needed and even asynchronously send and receive updates from the persistence store. An application architecture with in-memory data grid is shown below.
By using in-memory data grid, data moves closer to the application endpoints. This approach reduces the response times and can lower transactions times from a few seconds to fractions of a second. This way, an application can support extremely large numbers of concurrent transactions involving terabytes of operational data, providing a faster, more reliable transactional experience for customers. It is also a more modern scalable data management system than traditional RDBMS, able to elastically scale as demand increases.
Resilient Web Acceleration
With in-memory data grid like Apache Ignite, you can provide fault tolerance of your web application and accelerate your web application performance. Without changing any code, you can share session states between web applications through caches.
This above approach provides the highest level of high availability of a system and a customer experience. Since Ignite is an in-memory solution, the performance of the web session clustering and replication mechanism of user web sessions are very high.
Event Processing and Real-Time Analysis
Data tells the story of what's happing with your business on the background right now. With the IoT as a continuous data source, the opportunities to take advantage of the hot data is greater than ever. Traditional data management systems cannot process big data fast enough to notify the business of important events as they occur: such as online credit card fraud detection or risk calculation. Apache Ignite allows processing continuous never-ending streams of data in scalable and fault-tolerant fashion in-memory, rather than analyzing data after it's reached the database.
Not only does this enable you to correlate relationships and detect meaningful patterns from significantly more data that you can process it faster and much more efficiently; Apache Ignite in-memory data grid can manage a tremendous amount of incoming data and push notifications to the business application when changes occur with the server. The Apache Ignite continuous queries capability allows systems to quickly access a large amount of incoming never ending data and take action.
Microservices in Distributed Fashion
Microservice architecture has a number of benefits and enforces a level of modularity that in practice is extremely difficult to achieve with a monolithic code base. In-memory data grid like Apache Ignite can provide independent cache nodes to corresponding microservices in the same distributed cluster and gives you a few advantages over traditional approaches.
It allows you to use maximum uses of the data fabrics/grid resources. Services running on the in-memory cluster is much faster than the disk-based application server. Apache Ignite microservice based service grid provides automatically deploy any number of distributed service instance in the cluster.
Big Data Accelerator
Hadoop has been widely used for its ability to economically store and analyze large data sets and has long passed the point of being nascent technology. However, it’s batch scheduling overhead and disk-based data storage have made it unsuitable for use in analyzing live, real-time data in the production environment. One of the main factors that limit performance scaling of Hadoop and Map/Reduce is the fact that Hadoop relies on a file system that generates a lot of input/output (I/O) files. An alternative is to store the needed distributed data within the memory. Placing Map/Reduce in-memory with the data it needs eliminates file I/O latency.
Apache Ignite has offered a set of useful components allowing in-memory Hadoop job executing and file system operations. Apache Ignite Hadoop accelerator can automatically deploy all necessary executable programs and libraries for the execution of MapReduce across the JVMs, greatly reducing startup time down to milliseconds. This speeds up analysis by avoiding delays in accessing secondary storage. Also, because the execution engine is integrated with the in-memory data grid, key/value pairs hosted within the data grid can be efficiently read into the execution engine to minimize access time.
Cache as a Service
Data-driven applications that take too long to load are boring and frustrating to use. Four out of five online users will click away if a page stalls while loading. In-memory data grid can provide a common caching layer across the organization, which can allow multiple applications to access managed in-memory cache.
You can isolate the caching layer from the applications by separating the caching layer from the applications. Any applications (Java, .Net, C++) across the organization can store and read data from the cache. By using in-memory data grid as a service, it’s not necessary to build and deploy local caching infrastructure for each application. Applications can use Apache Ignite as cache-aside or write behind to their database or load data from the database into the cache. It eliminates the complexity of management of the hundred or more separate caching infrastructures.
These are some of the ways in-memory grids like Apache Ignite have served as an essential, architectural component for transforming the way businesses use their data to do business. We cover a lot of in-memory data grid use cases and application architecture in more depth through this book, from which portions of this article were taken.
Published at DZone with permission of Shamim Bhuiyan . See the original article here.
Opinions expressed by DZone contributors are their own.