Tarantool Queues (Part 3): The Art of Queue Parsing

DZone 's Guide to

Tarantool Queues (Part 3): The Art of Queue Parsing

Read this tutorial to learn more about how to achieve an implementation of a Tarantool-based instance with the tarantool-authman module.

· Database Zone ·
Free Resource

Image title

In our previous article, we used the tarantool-authman module to implement an authentication server. Tarantool-authman is quite good and has almost all of our desired functionality, but in complex distributed systems (including microservice ones), many subsystems must work asynchronously. So, if we implement a system with multiple services and a single authentication tool, the latter can become a bottleneck and will slow down the entire operation. Fortunately, queues provide a solution to our problem.

Our goal in this article is to achieve an implementation of an authentication server, i.e. a Tarantool-based instance, with the tarantool-authman module. To help this server cope with an inevitable flood of web-service requests, we’ll add a queue server made with Tarantool Queue.

Why Straightforward Queue Parsing Never Works

The queue server is used as follows:

Start server:

package.path = package.path .. ';/usr/local/lua/tarantool-queue/?.lua'
box.cfg {listen = 3301, slab_alloc_arena = 2}
queue = require 'queue'
box.queue = queue

Create a logical queue:

queue.create_tube('tube_name', 'fifo')

Parse the queue:

queue = Tarantool.Queue(host="localhost", port=3301)
while True:
task = queue.take(tube="tube_name")

The most straightforward and seemingly obvious solution to our problem is to have the queue server create a logical queue and fill it with jobs, while at the same time have the authentication server parse the queue, execute tasks, and delete them. However, with complex distributed systems, this approach won’t work for several reasons. First, the authentication server has to send it's results to a remote web service, which initiates tasks. In the case of a successful authentication, this remote needs to send back a positive result and, for example, a session ID, but otherwise needs to return a negative result. When registering a user, it needs to send an email with a confirmation code (a token) and report this to the remote web service — so it necessarily gives the user a code request window. Then, the web service initiates a job to complete the registration. So basically, data exchange must go in two directions, and jobs can vary a lot.

Hence, we need to create at least two logical queues — one consisting of the tasks sent to the authentication server, and the other including the results. Accordingly, the queue server also has to parse the results, perform actions, and provide responses to our web services. In reality, there will be even more queues because we have many job types, and in terms of both queue and authentication server optimization, it is wise to separate them. Moreover, when using high-load systems, it is ideal to address Tarantool in a single connection to optimize the use of computing resources. Thus, we naturally arrive at the necessity of asynchronous operation (at least in terms of queue parsing) for our complicated bundle and, therefore, the need to use asynchronous programming techniques.

API Unification and Security Problems

Lua is a powerful and efficient language that enables the creation of many cool things, but since our project requires an asynchronous approach to programming, some security issues arise when using it. If we execute our queue server/authentication server pairs simply as two Tarantool instances, the databases will be exposed and open to all digital “winds,” which is not good. In theory, such an approach is possible: there are Tarantool connectors for different programming languages, and external services are quite capable of accessing them (we mean the queue server). But, in addition to the obvious security issues (just to remind you, we are talking about a complex system, “scattered over the Internet”), we would need to link each web service (which uses authentication) to a Tarantool connector.

A more versatile and safe way involves the use of HTTP(S) and the implementation of a web API to interact with the authentication service. Ideally, with such an approach, a web service knows nothing about the authentication service internals except this very standardized API — it doesn’t care if there is a queue or not. For simplicity, we will use NGINX, already discussed in previous articles. Next, we will need more software to deal with our queues and tasks.

Choosing Tools
There are plenty of great programming languages and asynchronous frameworks out there. However, we will use Python because of its simplicity, its built-in library for asynchronous I/O, and it's multitude of useful third-party tools. Our complete ingredients will be as follows: Ubuntu 16.04 (but use any Linux distribution you’d like), Tarantool 1.7, Python 3, asyncio (the standard Python 3 library available out of the box and there is documentation), tarantool-queue, asynctnt (the Tarantool connector for asyncio), and asynctnt-queue (asyncio bindings library for the tarantool-queue package, integrated with the asynctnt module). We won’t go into the installation process for these, as it has been described more than once in our articles. Please refer to them as well as the documentation. So, the code to work with queues looks like this:

Tarantool config:

box.cfg {
listen = ''

box.once('v1', function()
box.schema.user.grant('guest', 'read,write,execute', 'universe')

queue = require('queue')
queue.create_tube('test_tube', 'fifottl')

Python code:

import asyncio
import asynctnt
import asynctnt_queue

async def run():
conn = asynctnt.Connection(host='', port=3301)
await conn.connect()

queue = asynctnt_queue.Queue(conn)
test_tube = queue.tube('test_tube')

# Add a task to queue
task = await test_tube.put({
'key': 'value'

print('Task id: {}'.format(task.task_id))
print('Task status: {}'.format(task.status))

# Retrieve a task from queue
task = await test_tube.take(1)

# ... do some work with task

await task.ack()
await conn.disconnect()

loop = asyncio.get_event_loop()

Using this tool, we can start parsing a plurality of logical queues within one process and one connection, and all of the jobs in our queue will be parsed swiftly. There won’t be timeouts, queue pulling, or other slowdowns. Terrific, right? Were we to use a synchronous approach, the code would be much more complicated and much less productive.


So far, we’ve discussed what queues are and why you need them, and we have examined an open-source tool for creating a queue server, an authentication server, as well as asynchronous I/O. In our next two articles, we will put the puzzle together and make a pair of authentication server/queue server combinations to process queues, as well as a unified web API for third-party web apps, so that the apps can work with our authentication service.

authentication, database, lua, python, queue management, queueing, tarantool

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}