Experience Using Azure Cosmos DB in a Commercial Project

Greetings, DZone readers! In this article, we will share with you our experience using the Azure Cosmos DB database service in a commercial project. We will tell you what the DB is for and the nuances we came across throughout development.

What Is Azure Cosmos DB?

Azure Cosmos DB is a commercial, globally distributed database service with a multi-model paradigm, provided as a PaaS solution. It is the next generation of Azure DocumentDB. The database was developed in 2017 at Microsoft Corporation with the participation of Dr.Sci. Leslie Lamport (winner of the Turing Award 2013 for a fundamental contribution to the theory of distributed systems, the developer of LaTex, the creator of the TLA + specification).

The main characteristics of Azure Cosmos DB are:

• Non-relational database;
• Storage of documents in the JSON format;
• Horizontal scaling with the ability to select geographic regions;

• Multi-model data paradigm: key-value, document, graph, family of columns;
• Low latency for 99% of queries: less than 10 ms for read operations and less than 15 ms for (indexed) write operations;
• Designed for high throughput;
• Ensures availability, consistency of data, delay at SLA level of 99.999%;
• Configurable throughput;
• Automatic replication (master-slave);
• Automatic data indexing;
• Configurable levels of consistency of data. Five different levels (Strong, Bounded Staleness, Session, Consistent Prefix, Eventual);

Notice the dependence of different levels of consistency on availability, performance, and consistency of data.

The graph is up - availability, performance; graph down - consistency

• For a convenient transition to Cosmos DB from your database, there are many APIs for accessing data: SQL, JavaScript, Gremlin, MongoDB, Cassandra, Azure Blob;
• Customizable firewall;
• Configurable DB size.

The Problem We Solved

Thousands of sensors (hereinafter - notifications) located around the world transmit information every few N seconds. These notifications should be stored in the database, and in the future, be able to be searched and displayed in the UI of the system operator.

Requirements from the customer:

• Use the Microsoft technology stack, including the Azure cloud;
• Throughput of 100 requests per second;
• The notifications do not have a clear structure and could be further expanded;
• For critical notifications, processing speed is important;
• High Fault Tolerancy of the system.

Based on customer requirements, we were ideally suited for a non-relational, globally distributed, reliable commercial database.

If you look at databases similar to Cosmos DB, you would find Amazon DynamoDB, Google Cloud Spanner. But Amazon DynamoDB is not globally distributed, and Google Cloud Spanner has fewer levels of consistency and types of data models (only a table view, a relational one).

For these reasons, we settled on Azure Cosmos DB. Azure Cosmos DB SDK for .NET was used to interact with the database since the backend was written in .NET.

The Nuances We Encountered

Database Management Tools

In order to start using the database, first of all, you need to select a tool to manage it. We used Azure Cosmos DB Data Explorer in the Azure portal and DocumentDbExplorer. Also, there is a utility Azure Storage Explorer.

Configuring Database Collections

In Cosmos DB, each database consists of collections and documents.

Customizable characteristics of the collection, which should be noted:

• Size of collection: fixed or unlimited;
• Bandwidth per request units per second RU/s (from 400 RU/s);

• Indexing policy (Including or excluding documents and paths to and from the index and from it, configuring various index types, setting up index update modes);

Example of a typical index:

{
  "id": "datas",
  "indexingPolicy": {
    "indexingMode": "consistent",
    "automatic": true,
    "includedPaths": [
      {
        "path": "/*",
        "indexes": [
          {
            "kind": "Range",
            "dataType": "Number",
            "precision": -1
          },
          {
            "kind": "Hash",
            "dataType": "String"
          },
          {
            "kind": "Spatial",
            "dataType": "Point"
          }
        ]
      }
    ],
    "excludedPaths": []
  }
}

To search for a substring for string fields, you need to use the Hash-index ("kind": "Hash").

Database Transactions

In the database, transactions are implemented at the stored procedures level (Execution of a stored procedure is an atomic operation). Stored procedures are written in JavaScript.

var helloWorldStoredProc = { 
	id: "helloWorld", 
	body: function () { 
		var context = getContext(); 
		var response = context.getResponse(); 
		response.setBody("Hello, World"); 
	} 
}

Database Change Channel

Change Feed listens for changes in the collection. When there are changes in the collection documents, the database throws out the event about the changes to all the subscribers of this channel.
We used Change Feed to track changes to the collection. When creating a channel, you must first create an auxiliary AUX collection, which coordinates the processing of the change channel for several work roles.

Database Restrictions:

• Absence of bulk operations (used stored procedures for mass deletion, updating of documents);
• No partial update of the document;
• There is no operation SKIP (complexity of the implementation of pagination). To implement the pagination in requests to receive notifications, we used RequestContinuation (reference to the last element as a result of issuance) and MaxItemCount (the number of items returned from the database). By default, results are returned in packets (no more than 100 items and no more than 1 MB in each package). The number of items returned can be increased to 1000 by using the MaxItemCount parameter.

Processing the 429th Database Error

When the collection Throughput reaches a maximum, the database starts to produce an error "429 Too Many Request". To process it, you can use the RetryOptions setting in the SDK, where MaxRetryAttemptsOnThrottledRequests is the number of attempts to execute the query, and MaxRetryWaitTimeInSeconds is the total time that connection attempts are attempted.

Forecasting the Cost of Using the Database

To predict the cost of using the database, we used the online calculator RU / s. In the base plan, 1 query unit for a 1 KB element corresponds to a simple GET command by reference to itself or the identifier of that element.

Resume

Azure Cosmos DB is easy to use, easily and flexibly configured through the Azure portal. A lot of APIs for accessing data allow you to quickly transition to Cosmos DB. You do not need to get the database administrator to maintain the database. Financial guarantees of SLA, global horizontal scaling make this database very attractive in the market. It is great for use in corporate and global applications that require high demands on fault tolerance and throughput. We at WaveAccess continue to use Cosmos DB in our projects.