Tarantool Queues (Part 5): Mission Complete

DZone 's Guide to

Tarantool Queues (Part 5): Mission Complete

In our final article in the Tarantool Queues series, we demonstrate how to set up an authentication server and how to structure Python code for queue processing.

· Database Zone ·
Free Resource

Image title

At this stage, we have a bundle that includes the NGINX/NGINX upstream module and a Tarantool instance with the Queue module. The bundle implements an API for remote web applications, which means that they will be able to interact with the authentication service via HTTP(S) and add their tasks to the queue. In this article, we will add the last elements required for our project: the authentication server itself and the app in Python to process the queue.

The Authentication Server (Tarantool+Tarantool-Authman)

For the authentication server, you will need to install the tarantool-authman module and run yet another Tarantool instance. We have already presented this bundle in our previous article, which you’re welcome to read. Note that our authentication service runs on a Mail.Ru Infra Ubuntu 16.04 virtual machine and the Tarantool repository is already connected. So, the installation is done with a single command:

$ sudo apt-get install tarantool-authman

Next, we need to make a file for our instance code in /etc/tarantool/instances.available/a1.lua:

box.cfg {
listen = 'localhost:3302'

box.once('grant', function()
box.schema.user.grant('guest', 'read,write,execute', 'universe')

local config = {
-- configuration table, check tarantool-authman documentation for details

auth = require('authman').api(config)

The instance will use port 3302, since we have the queue server on 3301. We just need to create a symlink for a1.lua in the /etc/tarantool/instances.enabled directory and the authentication server will be ready. In Ubuntu, it can be started with a single command:

$ sudo tarantoolctl start a1

Now we will work on its interaction with the other subsystems.

Processing Queue Tasks

To process queue tasks, we need Python 3, asyncio (a standard Python 3 library, available out of the box, see the documentation), the asynctnt library (the Tarantool connector for Python/asyncio), and the asynctnt-queue library (Python/asyncio bindings library for the Tarantool Queue package, integrated with the asynctnt module). Installing Python libraries requires the pip utility. Since I have two versions of Python (second and third) installed, I get the tool with the command:

$ sudo apt-get install python3-pip

(We will be writing code in Python 3, so the package manager is needed for the third version; note that in other distros both package and binary names may vary).

The Python packages are then installed with a single command:

$ pip3 install asynctnt asynctnt-queue

Now we can start developing for the queue analysis program. The code is quite short:

#!/usr/bin/env python3

import asyncio
import asynctnt
import asynctnt_queue

async def run():
conn_queue = asynctnt.Connection(host='', port=3301)
conn_auth = asynctnt.Connection(host='', port=3302)

await conn_queue.connect()
await conn_auth.connect()

queue = asynctnt_queue.Queue(conn_queue)
q1 = queue.tube('q1')

while True:
# Retrieve a task from queue
task = await q1.take(1)
if task:
if task.data['type'] == 1:
# ... do some work with task
print('Task data (registration): {}'.format(task.data))
result = await conn_auth.call('auth.registration', [task.data['login']])
print('Auth data (registration): {}'.format(result.body))

elif task.data['type']==2:
# ... do some work with task
print('Task data (complete): {}'.format(task.data))
result = await conn_auth.call('auth.complete_registration', [task.data['login'], task.data['token'], task.data['pass']])
print('Auth data (complete): {}'.format(result.body))

elif task.data['type']==3:
# ... do some work with task
print('Task data: {}'.format(task.data))

elif task.data['type']==4:
# ... do some work with task
print('Task data: {}'.format(task.data))

elif task.data['type']==5:
# ... do some work with task
print('Task data: {}'.format(task.data))

elif task.data['type']==6:
# ... do some work with task
print('Task data: {}'.format(task.data))

elif task.data['type']==7:
# ... do some work with task
print('Task data: {}'.format(task.data))

await task.ack()
print('Task status: {}'.format(task.status))

# await conn_queue.disconnect()
# await conn_auth.disconnect()

loop = asyncio.get_event_loop()

Thus, all the tasks added to the queue by third-party web apps and services will be processed and executed, and our authentication service is complete. Remember that external web applications interacting with the authentication service via HTTP(S) know nothing about Tarantool inside of it. To test it, let’s register a new user in the system (wget simulates requests from an external web application/service to our authentication service):

In another terminal, we have our Python code launched simultaneously. It monitors the queue, processes tasks, and writes the results to the console:

At Last!

As you can see, making an authentication service with Tarantool is pretty easy. During this series, we have shown how to accomplish it with ready-made modules. We made a queue server, an authentication server, and we set up NGINX interaction with Tarantool using the Tarantool NGINX upstream module. In addition, we learned how to use Python for Tarantool.

In our next series of articles, we will continue to develop our training project by turning it into a full-fledged API Gateway — an intelligent proxy server that can be used as a single entry-point for any microservice. This will solve for authentication/authorization, load balancing, installation constraints, and many other issues. It will also simplify the development of new API versions as well as the support for older API versions. So, stay tuned to get your smart proxy!

asyncio, authentication, database, python, queue management, queue messages, queueing theory, tarantool

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}