Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Develop an API Gateway Using Tarantool (Part 3): Queue Parsing With Python

DZone's Guide to

Develop an API Gateway Using Tarantool (Part 3): Queue Parsing With Python

In Part 3 of this API gateway tutorial, we'll learn about queue parsing with Python to build a base for URI parsing and a tool to request microservices.

· Microservices Zone ·
Free Resource

Learn how modern cloud architectures use of microservices has many advantages and enables developers to deliver business software in a CI/CD way.

This article begins the most interesting part of our experiment, as our smart proxy will learn to handle client HTTP requests. Let’s recall that we already have authentication by login and password, and we can add requests to a queue. So, it’s high time for queue parsing (let’s use Python)!

First, we will need a route registry for our services. However, we don’t need to keep the registry in the database and run a separate Tarantool instance for it because we’re only making a simple API Gateway for a distributed, but relatively small system. The number of services here is measured in the tens at most, so the overhead costs of addressing even a fast-working, non-relational DBMS to determine the routes seems excessive. It’s easier to store the registry directly in our queue parsing tool. On the other hand, it’s better not to endlessly change the code when we change system configurations. So let’s describe the registry in a JSON configuration file that will be read at startup.

At this step our API Gateway is mainly designed for authentication, so our test.json registry is simple too, although it makes everything clear:

{
   "auth":[
      {
         "method":"registration",
         "type":1,
         "params":[
            "login"
         ]
      },
      {
         "method":"complete_registration",
         "type":2,
         "params":[
            "login",
            "token",
            "pass"
         ]
      },
      {
         "method":"auth",
         "type":3,
         "params":[
            "login",
            "pass"
         ]
      },
      {
         "method":"set_profile",
         "type":4,
         "params":[
            "id",
            "profile"
         ]
      },
      {
         "method":"check_auth",
         "type":5,
         "params":[
            "session_id"
         ]
      },
      {
         "method":"drop_session",
         "type":6,
         "params":[
            "session_id"
         ]
      },
      {
         "method":"delete_user",
         "type":7,
         "params":[
            "id"
         ]
      }
   ],
   "api_gateway":{
      "method":"api_gateway",
      "params":[
         "token"
      ]
   }
}


As for the Tarantool instance files for both the queue and authentication servers, we will leave them unchanged from the previous articles in this series.

Now we are ready for queue parsing. We will be using two Python tools. To authenticate the service, we have created a kind of queue “sorter,” auth_api.py:

#!/usr/bin/python

import asyncio
import asynctnt
import asynctnt_queue
import json

async def run():

    with open("test.json") as json_file:
    json_data = json.load(json_file)

    conn_queue = asynctnt.Connection(host='127.0.0.1', port=3301)
    await conn_queue.connect()

    conn_auth = asynctnt.Connection(host='127.0.0.1', port=3302)
    await conn_auth.connect()

    queue = asynctnt_queue.Queue(conn_queue)

    tube = queue.tube('q1')

    while True:

       # Get task from the queue
       task = await tube.take()

       print('Task data: {}'.format(task.data))

       if task.data['type']:
           args = []
           method = ''
           for api in json_data['auth']:
               if api['type'] == task.data['type']:
                   method = 'auth.'+(api['method'])
                   for arg in api['params']:
                       args.append(task.data[arg])
           # Authentication operations
           result = await conn_auth.call(method, args)

       print('result {}'.format(result))

       # Mark the task as done
       if task:
         await task.ack()

    await conn_queue.disconnect()

loop = asyncio.get_event_loop()
loop.run_until_complete(run())


When started, this file reads the JSON configuration file, parses everything, and performs authentication-related tasks. You can check it with any HTTP connector; here we will use curl as it seems more suitable for this purpose than the wget we used before.

The API Gateway code at this stage is not 100% ready, although it is generally complete for queries not related to authentication. So far, api_gateway.py is just a stub that will be refined in the next article:

#!/usr/bin/python

import asyncio
import asynctnt
import asynctnt_queue
import json
import urllib3

def request(req_url, args):
    try:
        http = urllib3.PoolManager()
        encoded_data = json.dumps(args).encode('utf-8')
        r = http.request('POST', req_url,body=encoded_data,headers={'Content-Type': 'application/json'})

        return json.loads(r.data.decode('utf-8'))

    except urllib3.exceptions.NewConnectionError as e:
        return (e.code, json.loads(e.read()))


async def run():
    with open("test.json") as json_file:
        json_data=json.load(json_file)

    conn_queue=asynctnt.Connection(host='127.0.0.1', port=3301)
    await conn_queue.connect()

    queue=asynctnt_queue.Queue(conn_queue)

    tube=queue.tube('q2')

    while True:

        # Get task from the queue
        task=await tube.take()

        print('Task data: {}'.format(task.data))

        if task.data['query']:
            if task.data['token']:
                # Requests without authentication
                result=request('http://127.0.0.1:8081/api_gateway', task.data)

        print('result {}'.format(result))

        # Mark the task as done
        if task:
            await task.ack()

    await conn_queue.disconnect()


loop=asyncio.get_event_loop()
loop.run_until_complete(run())


Terrific! Now we have a base for URI parsing plus a tool to request microservices. In the next and last article in this series, we will make a more complex registry and we will implement the ability to request arbitrary microservices with a random API, to process compound queries, to load balance (when, for example, there are multiple instances of the same service running) and to return results to the client. Stay tuned!

Discover how to deploy pre-built sample microservices OR create simple microservices from scratch.

Topics:
tarantool ,api gateway ,tutorial ,microservices

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}