Couchbase .NET SDK 2.0 Development, Part 1

[This post was originally posted by Jeff Morris]

Hello folks, over the next couple of months we’ll be doing a series of blogs that will follow the development of the Couchbase .NET 2.0 SDK. In this post I will go over the high level architecture, some motivation and features you should expect, and what the overall goals are of the .NET SDK with respect to the other Couchbase clients: Node, PHP, Java, and C. Discussing the goals of the .NET SDK within the scope of the entire suite of Couchbase clients is an important step in unifying the API’s, so that developers have a consistent programming experience no matter which SDK they chose.

History and Motivation

First up, let’s talk about what are motivation is for re-writing the Couchbase .NET SDK. Note that for the most part, this is a complete rewrite from scratch…pretty much every component is being redesigned and redeveloped to better suit the needs of our customer and the user community at large. By now, most of us  have been warned that the single worst strategic mistake a software company can do, is rewrite working, functional code from scratch, but we here at Couchbase feel it’s the best decision given the history and state of the .NET 1.X SDK and the direction we want to go with the client.

A little history might be in order here. The current client is based off of the original Memcached .NET client: Enyim.Caching. It was written several years back and originally supported the Memcached Text protocol and eventually the Memcached Binary Protocol as well. In itself, it was designed to be a stand-alone API, not framework for creating other API’s. That being said, the current .NET Couchbase Client was built on top of a fork of the Enyim.Caching source; we forked because the original author, while a great supporter, no longer had the time needed to keep the project going. The Couchbase client added view support and other Couchbase Server specific features on top of the K/V stuff already supported by Enyim.Caching.

Since the client was originally based off a K/V based protocol, fitting in some of the Couchbase features has made for a bit of a thorny API. Furthermore, as more layers have been added the complexity has increased as well making supporting and maintaining the client more and more difficult. Additionally, since the client is really based off of .NET 3.5 (while we do build for 4.0, 3.5 is the LCD), we weren’t progressing with the current state-of-the-art .NET features like the powerful new Task (async/await) based asynchronous libraries/features and improvements to the networking API’s. Maintaining backwards compatibility and progressing the client is therefore impossible; we need to take a whole new direction.

Goals and Features

The goals/objectives of the Couchbase .NET 2.0 SDK are fairly simple:

• Unification and consistency of the programming interface across all Couchbase SDK platforms (Java, .NET, PHP, Node, C, etc)
• A flexible, test-driven design – extensibility at its core
• Highly configurable – but easy to get up and running with the default configuration
• Asynchronous I/O – a non-blocking model for network programming
• A simple and easy to use programming interface
• Minimalistic use of client/server connections
• Improved documentation

Some likely features include:

• Support for Cluster Carrier Configuration Publication (CCCP) – a new way of updating the clients configuration based on the clusters current state via a pull mechanism
• Task-based Asynchronous Pattern (TAP) support for K/V and View operations
• First class N1QL support – N1QL is the awesome new SQL-like query language in Couchbase
• A LINQ provider implementation over the core API
• Integration points with other .NET technologies like ASP.NET and Frameworks for quickly developing data-driven applications

High-Level Architecture

Over the last couple years, the SDK team at Couchbase have developed several Couchbase Server clients on different platforms and frameworks: .NET, Java, Node.js, etc. As each of these clients have been developed and evolved with changes in the Server, several patterns have emerged for developing a cluster aware client, or more specifically in Couchbase’s case, a “smart” client. In the goal of “unifying the programming interface”, we implement these patterns within each client. We will be discussing those patterns in detail in later posts, but first I would like to show a high-level view of the client components:

Couchbase Client Components

These components make up a Couchbase client:

• IO – Resource management and general network programming access
• Configuration – Client and Server configuration
• Operations – Key/Value and View operations on JSON documents: Get, Set, etc.
• Cluster Map – Manages active nodes and where keys are stored across the cluster
• Logging and Instrumentation – General logging, tracing and monitoring of client activity

Next Up

Well, that’s it for now. Next up we will be going over the design and development of one portion of the Configuration component: Server Configuration. The Server Configuration drives the state of the client which parallels the state of the cluster. A change in the cluster generally, means a change in the client state as well, which brings with it various complexity and issues which must be addressed to keep things stable.