Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Intro to API Caching With Zato 3.0

DZone's Guide to

Intro to API Caching With Zato 3.0

The upcoming Zato 3.0 release will bring API caching functionality. Learn about this feature and how it is used in this primer.

· Integration Zone ·
Free Resource

SnapLogic is the leading self-service enterprise-grade integration platform. Download the 2018 GartnerMagic Quadrant for Enterprise iPaaS or play around on the platform, risk free, for 30 days.

One of many features that the upcoming Zato 3.0 release will bring is API caching - this post provides an overview of the functionality that is already available if one installs Zato directly from the source code.

What's New

  • In-RAM caches accessible to Python-based services and external clients (including REST and other format or protocols)
  • A convenient set of practical cache commands based on real-world needs
  • GUI and cache contents search
  • HTTP channel caching
  • Integration with Memcached

Built-in Caches

In-RAM caches are perfect for rarely changing yet frequently read business data. For instance:

  • A customer's products, stored in a remote SQL database, will change infrequently but may be read very often.
  • A user's permissions may be read from multiple databases and the process may take several seconds but once read, they will not change much and fast access is of primary importance.

Each Zato cluster has a default cache and there can be multiple independent caches for different purposes each with its own configuration.

In v3.0, there is no persistent storage for caches and synchronization between servers in a Zato cluster is performed in the background. Both aspects will be extended in future releases.

Real-World Commands

Built-in caches offer all the standard CRUD commands operating on cache entries but they come in several versions, each of which is meant to facilitate operations in real-world applications.

All of the commands are available to Python services and external API clients through dedicated REST endpoints.

More commands will be added in the future, based on feedback from users employing caches in projects.

Command Description
get Returns an entry matching input key
get-by-prefix Returns all entries whose keys start with a given prefix
get-by-suffix Returns all entries whose keys end with a given suffix
get-by-regex Returns all entries whose keys match a regular expression
get-contains Returns entries with keys that contain a single input substring
get-contains-all Returns entries with keys that contain all input substrings
get-contains-any Returns entries with keys that contain at least one input substring
get-not-contains Returns entries with keys that do not contain the input substring
set Sets input value for a given key
set-by-prefix Sets input value for all keys starting with a given prefix
set-by-suffix Sets input value for all keys ending with a given suffix
set-by-regex Sets input value for all keys matching a regular expression
set-contains Sets input value for all keys that contain a single input substring
set-contains-all Sets input value for keys that contain all input substrings
set-contains-any Sets input value for keys that contain at least one of input substrings
set-not-contains Sets input value for keys that do not contain the input substring
delete Deletes a given key
delete-by-prefix Deletes all keys starting with a given prefix
delete-by-suffix Deletes all keys ending with a given suffix
delete-by-regex Deletes all keys matching a regular expression
delete-contains Deletes all keys that contain a single input substring
delete-contains-all Deletes keys that contain all input substrings
delete-contains-any Deletes keys that contain at least one of input substrings
delete-not-contains Deletes keys that do not contain the input substring
expire Sets expiration for a given key
expire-by-prefix Sets expiration for all keys starting with a given prefix
expire-by-suffix Sets expiration for all keys ending with a given suffix
expire-by-regex Sets expiration for all keys matching a regular expression
expire-contains Sets expiration for all keys that contain a single input substring
expire-contains-all Sets expiration for keys that contain all input substrings
expire-contains-any Sets expiration for keys that contain at least one of input substrings
expire-not-contains Sets expiration for keys that do not contain the input substring
keys Returns all keys in the cache
values Returns all values in the cache
items Returns all items (entries) in the cache
clear Clears out the cache completely

Additionally, Python code can access dict-like methods .iterkeys, .itervalues and .iteritems that return iterators over keys or values without building result lists upfront.

Cache Configuration

Each cache can have its own policy that is best suited to a given purpose:

  • Cache size upon reaching of which the least commonly used entries will be evicted
  • Max item size - values that are too big will be rejected
  • Whether a given item's expiration time should be extended on each get or set operation

ScreenshotsScreenshots

GUI

As with other Zato features, a GUI in web-admin lets one easily manage cache definitions, add new or update existing entries from the browser, and look up items by keys, values or both.

ScreenshotsScreenshots

Auto-Caching in HTTP Channels

Caching of responses, either REST or SOAP, is such a common activity that it is available under one click for all HTTP channels.

When creating a new channel or updating an existing one, pick a definition of cache that should hold responses for that channel and servers will automatically cache them on the first request.

On subsequent invocations, all responses from that channel will be served from cache until underlying keys expire or are deleted.

Screenshots

Usage From Python

The self.cache object is an entry point to all the caches from a service's perspective. For instance, to add new entries:

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

# stdlib
from time import sleep

# Zato
from zato.server.service import Service

class CacheDemo(Service):
    """ A demo service that uses built-in caches.
    """
    def get_data(self):
        """ Returns data from external systems.
        """
        # Assume this takes a long time to return, e.g. several remote calls
        sleep(2)

        return 'MyData'

    def handle(self):

        # Get data from external resources
        data = self.get_data()

        # Store it in default cache
        self.cache.default.set('data', data)

        # Or, store it in an explicitly named cache
        cache = self.cache.get_cache('builtin', 'Customer products')
        cache.set('cust-123-data', data)


Both entries can be now found in web-admin:

ScreenshotsScreenshots

REST API

All cache commands are available through REST API calls. Here, set and get are used for illustrative purposes.

Note how input can be sent in both body and query string and that the last get command obtains metadata in addition to actual business value.

$ curl -XPOST localhost:11223/zato/cache -d '{"key":"my-key", "value":"my-value"}'
$
$ curl -XGET localhost:11223/zato/cache?key=my-key
{"value": "my-value"}
$
$ curl -XGET localhost:11223/zato/cache -d '{"key":"my-key"}'
{"value": "my-value"}
$ 
$ curl -XGET localhost:11223/zato/cache -d '{"key":"my-key", "details":true}'
{"hits": 4, "key": "my-key", "last_read": "2017-12-18T11:57:52.202461",
"value": "my-value", "expiry": null, "prev_write": "2017-12-18T11:56:57.714813",
"prev_read": "2017-12-18T11:57:47.225827", "expires_at": null,
"last_write": "2017-12-18T11:57:43.200757", "position": 0}
$ 


Summary

API caching in Zato 3.0 is a new feature, based on practical needs, that without a doubt will come in handy in many integration scenarios that require efficient and convenient access to data cached in RAM.

Watch this space for future posts describing in detail commands, configuration options, response hooks or runtime usage statistics API.

Download A Buyer's Guide to Application and Data Integration, your one-stop-shop for research, checklists, and explanations for an application and data integration solution.

Topics:
api ,caching ,python ,zato ,soa ,rest ,integration

Published at DZone with permission of

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}