In-Memory Database Architecture: Ten Years of Experience Summarized (Part 2)
Read this article for part 2 of the discussion of the architecture of Tarantool DB to showcase the benefits of living in RAM.
Join the DZone community and get the full member experience.Join For Free
An in-memory database is not a new concept. However, it is associated too closely with terms like "cache" and "non-persistent". In this article, I want to challenge these ideas. In-memory solutions have much wider use cases and offer higher reliability than it would seem at a first glance.
I want to talk about the architectural principles of in-memory databases, and how to take the best of the "in-memory world"— incredible performance — without losing the benefits of disk-based relational systems. First of all, how to ensure data safety.
This sums up ten years of experience with in-memory solutions in one place. At the same time, the reader doesn't need to have as much experience to benefit from this article, a basic familiarity with IT is enough.
In part one of this two-part article, we covered:
- Development history
- Tarantool today
- How does the core work
- Fibers and cooperative multi-tasking
This is the second part of the story.
- Database functionality
- Comparison to other systems
- In-memory platforms
- Relational databases
- Key-value databases
- Document-oriented databases
- Column-oriented databases
- Use cases
- When not to use
- When to use
My name is Vladimir Perepelitsa, also known as Mons Anderson. I am a Tarantool architect and product manager. I’ve been using it in production for many years now, for example, in the creation of S3 compatible object storage. So I know it inside and out.
To understand a technology, it is useful to take a look at its history. I will summarise what Tarantool was like at the beginning, what it has gone through, and what it is now. Then I will compare it to other databases; review its functionality; learn how it can work via a network, and see what is in the ecosystem around.
This example will let you know what benefits you can get from in-memory solutions. You will know how to use them without loss of reliability, scalability and usability.
We use tuples for storing data. These are arrays of untyped data. Tuples are joined in spaces. Space is essentially a collection of tuples. The equivalent in SQL world is a table.
There are two storage engines in Tarantool. Different spaces may be stored in memory or on disk. A primary index is always required for working with data. If only a primary index is created, Tarantool will look like a key value.
But there can be many such indexes. Indexes can be composite. They can have several fields. Selection by a partial match with the index is possible. We can proceed by index; that is, go around consecutively by an iterator.
There are indexes of different types. Tarantool uses B+*tree by default. There are also hash, bitmap, rtree, functional indexes, and JSON path indexes. With such diversity, we can use Tarantool quite successfully where the use of relational databases is appropriate.
Tarantool also has an ACID transactions mechanism. Arrangements for single-threaded access to data enable us to achieve ‘serializable’ isolation level. When we call Arena, we can write to or read from it, or modify data. All that happens is done consecutively and exclusively in one thread.
Two fibers cannot be executed in parallel. As far as interactive transactions are concerned, there is a separate MVCC engine. It makes it possible to execute interactive transactions in serializable mode; however, potential conflicts between transactions will need to be additionally handled.
Apart from the Lua access engine, Tarantool has SQL. We have often used Tarantool as a relational database. We realized that we designed the database according to relational principles.
We used spaces where SQL used tables. That is, each row is represented by a tuple. We have defined a schema for our spaces. It became clear to us that we can take any SQL engine, and just map primitives and execute SQL on top of Tarantool.
In Tarantool, we can invoke SQL from Lua. We can either use SQL directly or call what was defined in Lua from SQL.
SQL is a complementary mechanism, but it is up to you whether to use it. It’s a fairly good accessory that expands Tarantool applications.
Data storage primitives
- tuple (row)
- space (table) — a collection of tuples
- memtx — all of of data fits into memory, there is a copy on disk
- vinyl — data is stored on disk, may exceed memory size
- primary index
- can be numerous
- types of indexes
- tree (B⁺*)
- json path
- Serializable (No-yield)
- Interactive (MVCC)
SQL & Lua
- TABLE: space
- ROW: tuple
- Schema: space format
- Lua → SQL: box.execute
Comparison To Other Systems
Let’s compare Tarantool to other systems to understand its position in the world of databases. There are many ways to do this, but I want to focus on four key groups:
- In-memory platforms
- Relational databases
- Key-value solutions
- Document-oriented systems
I want to focus on GridGain, GigaSpaces, Redis Enterprise, Hazelcast, and Tarantool.
What are the similarities?: In-memory engine, in-memory database, plus some application runtime. They allow building cluster systems in a flexible way to accommodate different data volumes.
In particular, this allows for usage as a Data Grid. These platforms are aimed at business objectives. Every grid and every in-memory platform is based on its own architecture, yet they belong to the same class. Also, different platforms comprise different sets of tools, because each is aimed at its own segment.
Tarantool is a general-purpose platform that is not tied to any particular segment. This extends capabilities and the range of business scenarios that can be managed.
Let's now compare the in-memory engine of the Tarantool database to MySQL and PostgreSQL. This allows positioning the engine itself separately from the application server and platform.
Tarantool is like relational databases because it stores data in tabular form (in tuples and spaces). Indexes are built to data, the same as in relational databases. It is possible to define a schema in Tarantool, and there is even SQL to work with data.
But it is the SQL schema that makes Tarantool different from classic relational databases. Although SQL is present, its use may be omitted if desired. It is not the main tool for interaction with data.
Tarantool schema is not strict. You can define it only for some subset of your data.
In conventional relational databases, a table in memory is non-persistent storage that is used for some quick transactions. In Tarantool, the whole volume of data fits into memory, is served from memory, and is at the same time secure and persistent.
This is so important that it should be repeated: Tarantool stores the whole data set in memory, and the data is securely saved to disk at the same time.
The next class to compare to is key-value: Memcached, Redis, and Aerospike. What are the similarities? Tarantool can operate in key-value mode, and strictly one index can be used. In this case, Tarantool behaves like classic key-value storage.
For example, Tarantool can be used as a drop-in substitute for Memcached. There is a module that implements the corresponding protocol, and in this case, there is a complete simulation of Memcached.
Tarantool is similar to Redis by its in-memory architecture, it just has a different style of data description. Where Redis is applicable due to architectural scenarios, Tarantool can be used too. A more detailed comparison is provided here.
The difference of Tarantool from key-value databases is in the presence of secondary indexes, transactions, iterators, and other things characteristic of relational databases.
The fourth category to discuss here is document-oriented databases. The most vivid example here is MongoDB. Documents can be stored in Tarantool too. This is why it is possible to say that Tarantool in a way is a document-oriented database, among other things.
Tarantool’s internal storage format is msgpack, which is a kind of binary JSON. It is nearly equivalent to the format used in Mongo, BSON, and is compact. It reflects the same types of data. Furthermore, you can index the contents of the documents.
Tarantool also comes together with the Avro Schema library. It enables the decomposition of documents with regular structure into rows and storing the rows in a database.
However, Tarantool was not originally intended as a document-oriented database. This is a bonus, and a feature to store a part of data as a document. Therefore, its indexation mechanisms are somewhat weaker than Mongo’s.
Sometimes there are also questions about column databases. The answer is simple: Tarantool is not a column database (who would have thought). Scenarios which are good for column databases are not suitable for Tarantool. But they complement each other well.
Many of you may know ClickHouse, which is an excellent analytical solution and is a column database. Notably, ClickHouse doesn’t handle microtransactions quite well. It will not reach its maximum capacity when many small transactions are sent to it; it’s best to send data in batches.
As to Tarantool, microtransactions can be and should be sent to it, as it can accumulate them. Since it has various connectors, it can build up such transactions and send them to storage like ClickHouse in a batch. Perfect match!
Use Cases: When Not To Use
Let’s start with cases when you should not use Tarantool. The main scenario is analysis, also called OLAP, including the use of SQL.
The reasons are simple enough: Tarantool is intended as a single-thread application. It has no data access locks. However, while a thread executes a long SQL, no other operation will be possible.
Therefore, analytical databases usually use multi-thread data access. In this case, computations can be made in individual threads. In the case of Tarantool, a single thread works faster than in many other solutions. But there is only one thread, and it is not possible to work with data from several threads.
If, however, you want to build some pre-computed analysis, you will know that you’re going to need some specific cumulative data. You have a data flow, and you can say right away that you need some counters. Such pre-computed analysis can be built well with Tarantool.
Use Cases: When To Use
The main usage scenario stems from its historical original purpose. There are many small transactions.
These can be sessions, user profiles, and everything that has grown from it over time. For instance, Tarantool is frequently used as vector storage close to Machine Learning because it is convenient to use it for this. It can take up the role of high-load counters through which all traffic flows, or anti-Bruteforce systems.
Bad Use Examples
- Analysis (OLAP)
- Incl. with use of SQL
Good Use Examples
- High-frequency microtransactions (OLTP)
- User profiles
- Counters and attributes
- Cache-proxy to data
- Queue brokers
Tarantool is persistent and can work with many other systems. Therefore, it is used as a cache proxy to legacy systems. These can be heavy and complex systems, both write true proxy and write-behind proxy.
Furthermore, Tarantool’s architecture, availability of fibers, and features to write complex applications make it a good tool for writing queues. I personally know 6 queue implementations; some are available on GitHub, others are in closed repositories or in some projects.
The main reason for that is guaranteed low latency for access. When you are inside Tarantool and go for some data, you provide it from memory. You have quick competitive access to data. In this case, you can build hybrid applications which run really close to data.
Until the next time, goodbye!
Opinions expressed by DZone contributors are their own.