Over a million developers have joined DZone.

Tarantool Queues (Part 4): Queue Server API-ified

DZone's Guide to

Tarantool Queues (Part 4): Queue Server API-ified

Read this article, which includes a tutorial and explains how to write a more complicated queue server in Lua and how to create an API.

· Database Zone ·
Free Resource

Discover Tarantool's unique features which include powerful stored procedures, SQL support, smart cache, and the speed of 1 million ACID transactions on a single CPU core!

Image title

Let’s summarize our previous articles in this series on Tarantool Queues: we have an authentication service, which is an NGINX web server complete with the Tarantool NGINX upstream module. This implements an API for remote web services and it interacts with the queue server, which is a Tarantool instance with the Tarantool Queue package. In addition, there is another Tarantool instance with the tarantool-authman module. The latter is just our authentication server. For bilateral interaction between the queue server and the authentication server, we will do some coding in Python using asynchronous programming techniques. This way we can parse our queues, perform tasks, and so on.

Getting the Queue Server Ready (Tarantool + Tarantool Queue)

We will use a Mail.Ru Ubuntu 16.04 virtual machine, Tarantool 1.7, and the Tarantool Queue package.

In Ubuntu, you can easily install the two packages:

$ sudo apt-get install tarantool
$ sudo apt-get install tarantool-queue

Assuming that the software is successfully installed, let’s write our queue server instance in Lua. In the previous articles in the series, we used simple examples, but today, we will write more complicated code. For our instance to start automatically, we'll put a q1.lua file in the /etc/tarantool/instances.available directory plus create a symlink to it in the /etc/tarantool/instances.enabled directory (your distribution commands and path may differ somewhat).

So here is q1.lua:

-- Initiate an instance and create the queue if it does not exist yet

box.cfg { 
listen = 'localhost:3301' 

queue = require 'queue' 
box.queue = queue

box.once('grant', function()
box.schema.user.grant('guest', 'read,write,execute', 'universe') 

q1 = queue.create_tube('q1', 'fifottl', { if_not_exists = true })

-- Make stored procedures for our API

--a. Register a new user
function registration(email)
return q1:put( { type = 1, login = email }, { ttl = 60 } )

-- b. Complete the registration with confirmation code
function complete_registration(email, code, password)
return q1:put( { type = 2, login = email, token = code, pass = password }, { ttl = 60 } )

--c. Pass authentication
function auth(email, password)
return q1:put( { type = 3, login = email, pass = password }, { ttl = 60 } )

--d. Set the account profile
function set_profile(user_id, profile_table)
return q1:put( { type = 4, id=user_id, profile = profile_table }, { ttl = 60 } )

-- e. Get the user data through the session ID
function check_auth(session)
return q1:put( { type = 5, session_id = session }, { ttl = 60 } )

-- f. Kill the session
function drop_session(session)
return q1:put( { type = 6, session_id = session }, { ttl = 60 } )

--g. Delete the account
function delete_user(user_id)
return q1:put( { type = 7, id=user_id }, { ttl = 60 } )

Next, let’s run the instance:

$ tarantoolctl start q1

Great! The queue server is ready. For testing, you can, for example, connect to it and call a stored procedure in interactive mode:

$ tarantoolctl enter q1

At this stage, we no longer need the interactive mode for our queue server, we’ll work over HTTP(S) through a simple API. For this, we’ll make a few functions to add tasks in the queue, then we'll call them via requests sent through HTTP(S). The queue type here is fifottl, as fifo just won’t do because the task must be done in a reasonable amount of time, or the remote service will receive an error. In other words, the task life must be limited.

Creating the API (NGINX + the Tarantool NGINX Upstream Module)

This bundle has been described in our article RESTful Services Made Simple: How to Get Started in Three Steps, so you’re welcome to refer to the information there. I only note that dynamic plugins as a production-ready solution appeared in NGINX relatively recently, so developers still recommend installing NGINX with the Tarantool NGINX upstream module from source code. This process is both simple and well described. There is another non-canonical option—you may use the finished Docker image provided by Mail.Ru.

So, now that the software is installed on the server, our aim is to help the remote web apps/services send requests to HTTP(S) and add tasks to the queue. So let’s set up our nginx.conf file:

events {
worker_connections 1024;

http {
# Adding Tarantool as a backend
upstream backend {
# Tarantool hosts

# Web server running on port 8081
server {
location /api {
# Module on
tnt_pass backend;

After starting NGINX, you can call the stored procedure (e.g.“registration”) through any HTTP connector. Just send an HTTP POST request to and add JSON to its body. It looks like this when using wget:

$ wget --post-data '{"method":"registration","params":["test@mail.ru"],"id":1}'


At this stage, we have the web server plus the queue server bundle ready, so our authentication service is almost operational. Remote web-based apps and services can add tasks to the queue, calling Tarantool stored procedures. Since we’re only testing, web apps interact with the machine via NGINX HTTP, though in practice, you will need to use encryption. Let the HTTPS setting be your homework :).

In our next and final article, we will run the authentication server, which is another Tarantool instance with the tarantool-authman module, and we will write a program in Python to process and execute tasks. There is still one tiny problem, though: all parts of the service are asynchronous, so after placing a task in the queue, the remote application cannot immediately see the result. We can solve this problem in different ways, so we will supplement the API and do other interesting things in the next article. At the same time, we’ll learn the techniques of asynchronous programming in Python. Stay tuned, more fun to come!

Discover Tarantool's unique features such as powerful stored procedures, SQL support, smart cache, and the speed of 1 million ACID transactions on a single CPU.

queue management ,queue messages ,queue theory ,tarantool ,database ,nginx

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}