Functional and Integration Testing (FIT) Framework

We’ll start out the blog by introducing you to high-level architectural insight. Then, we’ll walk you through the development of the framework.

We will go through the various issues involved in testing Couchbase transaction SDKs and then discuss their resolutions. Relevant examples will also be used to show how they have affected the development of the framework. Please note that not all technical details of the framework are mentioned in this blog but it definitely attempts to give a holistic picture.

Couchbase offers transactions in multiple SDKs: Java, Dotnet, and CXX for now and with a plan to support Golang, Python, Node.js, PHP, and Ruby SDKs in near future. Testing SDKs that offer the same functionality would pose multiple problems during test automation. Test automation redundancy is the first one that would come to everyone’s mind. Apart from redundancy, we also have to ensure all SDKs have similar implementations of Couchbase transactions. For example, error handling is done exactly the same by all SDKs. These are just a couple of problems. With a major focus on transactions, this blog will provide various issues we would face while testing multiple SDKs and how we at Couchbase have solved them.

Introduction to Couchbase Transactions

Distributed ACID transactions ensure that when multiple documents are needed to be modified then only the successful modification of all justifies the modification of any, either all the modifications do occur successfully, or none of them occurs. Couchbase compliance with the ACID properties can be found here.

Transactions in Distributed Environments

Single node cluster: Couchbase transactions work on multi-node as well as single-node clusters. However, the cluster configuration should be supported by Couchbase.

Transactions support for N1QL queries: Ensure at least one of the nodes in the cluster has a query service. 

Couchbase Transactions SDK Testing

During the design phase of the framework, a deep analysis of the test plan and its automation posed us with multiple challenges. Below are a few of the major challenges and their resolutions. Going forward, we will discuss the problem and its solution and how it shaped the development progress of the framework.

Problem 1: Redundancy Problem

At Couchbase, we currently support transactions in three different SDKs: Java, Dotnet, and CXX. In near future, we will be supporting a few more SDKs including Golang. This clearly provides the QE with a redundancy problem, i.e. we might have to automate the same test case multiple times once for each SDK. 

Resolution: Each test case can be classified into three main parts:

1. Test preparation, e.g.: test data, test infrastructure, etc. 
2. Test execution, e.g.: transaction CRUD operations. 
3. Result validation.

A closer look at these three parts reveals that the actual SDK testing is involved only in the test execution phase while test preparation and result validation actually are independent of the SDK, i.e. it does not really matter which SDK is used. This led us to design a framework that consists of two parts: driver and performer. The driver takes care of test preparation and result validation while the performer does the test execution. The driver drives the test execution but only abstractly (we will learn more about this below) and issues commands to the performer. The performer acts on these commands and performs the actual test execution.

The FIT framework is designed in a client-server model where the driver acts as a client and the performer as a server.

Driver: Consists of all the tests preparation and result validation. All tests are the classic JUnit tests and can be executed either as a single individual test or as a specific test suite or entire tests suite. All the tests are written once and only one time. These tests can be reused for all the SDKs.

Performer: This is a simple application written once for each SDK. 

Protocol: Inside a driver, each test is molded in the form of a Java object and sent to the gRPC layer. The gRPC protocol does the work of converting this Java object into a language-specific test object and sends it to the performer. The performer gets this test object, reads the instructions, and executes the required transaction operations. Once the transaction is completed, the performer sends the result back to the driver via gRPC protocol

Once the driver receives the result object, it proceeds with the result validation. 

Test development process: Now that we have a top-level idea of how a driver and a performer operate inside of the FIT framework, let us see the technical aspect of it and how they interact with each other using a few simple example tests.

Example 1

Testing transactions with a single operation: the basic “replace” operation.

Driver code: 

   @Test

   public void oneUpdateCommitted() {

       collection.upsert(docId, initial);   // Test Preparation

       TransactionResult result = TransactionBuilder.create(shared)

           .replace(docId, updated)

           .sendToPerformer();      // Test Execution

           //Result validation

       assertCompletedInSingleAttempt(shared, collection, result);      

       assertDocExistsAndNotInTransactionAndContentEquals(collection, docId, updated);

   }

As you can see, all the tests are always written only once and as JUnit tests.

Test preparation and result validation are independent of SDK, hence done in the JUnit test itself. 

However, the test execution part is done in an abstract way. On the top, it will look like it’s executed in the driver itself. But it engages in distributed computing following the remote procedure call. The whole test is converted into a Java object, i.e. the TransactionBuilder object of our FIT framework, and then it is sent to the performer via gRPC layer using the sendToPerfomer method. 

In this example, where we are trying to test the transaction replace operation, we create a Java object that will have these details:

1. Document ID on which the transaction is supposed to execute
2. Transaction operation, in this case, is “replace”
3. Updated value, i.e. new value which we want the transaction to impose on the doc

Once you create such a Java object, the sendToPerformer invokes the gRPC and sends it to the performer.

Please refer to Java performer code: basicPerformer.

So, in the first step, the performer reads the test object and checks for the operation which it needs to execute. In our example, since it’s a replace operation, op.hasReplace() will return true and op.hasInsert(), op.hasRemove(), etc., will return false.

Inside the replace code block, the performer retrieves the document and the new content for the document. Once all the relevant information is retrieved, the performer executes the transaction, i.e. ctx.replace() operation.

Once the transaction is successfully executed, the result is sent back to the driver and the driver then similar to the performer retrieves the relevant information from the result object and performs the result validation.

Examples of functionality tested: This feature of the framework helped us in testing the transactions SDK not just for the doc content but also the transaction metadata, i.e. expected metadata is present wherever necessary and metadata is removed wherever necessary.

Now that we have some technical insight into the FIT framework, let’s get into a little more detail:

Example 2 

Below is driver code containing testing transactions with more than one operation:

   @Test

    void insertReplaceTest() {

        collection.upsert(docId2, initial); //Test preparation

        TransactionResult result = TransactionBuilder.create(shared)

                .insert(docId1, initial)

                .replace(docId2, updated)

                .sendToPerformer();  //Actual Test Execution

        //Result validation

        assertCompletedInSingleAttempt(shared, collection, result);      

        assertDocExistsAndNotInTransactionAndContentEquals(collection, docId1, initial);

        assertDocExistsAndNotInTransactionAndContentEquals(collection, docId2, updated);

    }

In this test, the transaction executes insert on docId1 and replace on docId2. So we have to add “insert” and “replace” into the test object and send all the relevant information to the performer

Please refer to Java performer code: performerSupportsTwoOps.

Since we have insert on the performer op, insert will return true and the performer retrieves the required information and performs the insert. Then op.replace() will return true and the performer will execute the replace operations and return the result back to the driver.

Examples of functionality tested: Multiple transaction operations on different documents were tested. Initially, we did not support all valid multiple transaction operations on the same doc in the same transaction. When this functionality was added, we could test it with this feature of the framework. Other tests like transactions maintain ACID features even when one of its operations fails and tests related to expiry were tested well with this support.

Both above examples are positive scenarios. Now let's come to negative scenarios i.e error and exception handling. These errors/exceptions are SDK specific so they need to be handled by the performer. So the driver needs to tell the performer what error/exception to except and the performer needs to do this validation.

Problem 2: Error Verification

1. For different failures, the transaction should understand the cause and throw the relevant error/exceptions. So we had to not only test the functionality of the transactions but also the error codes and exceptions thrown by them.
2. Transaction exception handling is different for each error/exception. For example, the document not found with an exception should be handled differently than some transient exceptions.
3. Even for the same exception, the point of occurrence will cause it to be handled differently. For example, the write-write conflict for insert/replace is handled differently than the replace/remove operations.

Resolution: The driver should send failure points, errors to be induced, and the expected cause and exceptions to the performer. The performer will read the failure point and error and induce them using hooks. Later it will ensure that the cause and exception codes from the transaction are exactly the same as those sent by the driver.

Hooks are internal Couchbase implementations that help to test failure scenarios. In our example below we are just trying to create an expiry before inserting a document

If either the exception is not thrown or an incorrect expectation is thrown, the performer fails the tests and sends the failure in the result object to the driver. The driver reads this result object and gives out the expected and the actual failure as output.

Example 3

Now we will test negative case scenarios.

Driver code:

@Test

    void expiryDuringFirstOpInTransactionEntersExpiryOvertime() {

        String docId = TestUtils.docId(collection, 0);

        TransactionResult result = TransactionBuilder.create(shared)

                .injectExpiryAtPoint(StagePoints.HOOK_INSERT)

                .insert(docId, updated, EXPECT_FAIL_EXPIRY)

                .sendToPerformer();

        ResultValidator.assertNotStarted(collection, result);

        DocValidator.assertDocDoesNotExist(collection, docId);

        assertEquals(TransactionException.EXCEPTION_EXPIRED, result.getException());

    }

So in this test, the driver is telling the performer to execute insert and then to expect the transaction to expire during this insert operation. We send the code EXPECT_FAIL_EXPIRY to convey this to the performer.

Please refer to the Java performer code: performerSupportsErrorHandling. 

Examples of functionality tested: All error/exception handling and error codes were tested. Also helped us to validate that all SDK's handle them in the very same manner. 

Problem 3: Version Management

We have to test different library versions of transactions and the newer versions would have new features not available in the previous versions. So the test framework has to understand which feature is not supported and avoid running those tests. 

Resolution: We have used the JUnit5 condition test execution extensions. Each test suite is annotated with a @IgnoreWhen condition. Before the driver starts executing any tests, it will contact the performer and get its version and all the functionalities supported by it. The IgnoreWhen on the driver uses this information and executes a test only when all the conditions given to it are satisfied.

Please refer to the Java driver code: driverSupportsVersionManagement.

Examples of functionality tested: We tested different versions of the same SDK, such as Java 1.1.0 vs. 1.1.0.  This helped us in the test-driven development, as well. The SDK that developed a feature a bit later than other SDKs could use this functionality to disable the tests and run later, enabling them once the feature is implemented.

Problem 4: Multiple Performers — Parallel Transactions

Transactions can be executed in parallel. Couchbase transactions confirm the isolation model, such as when two or more transactions are executed on the same set of documents, they should not lead to dirty writes/reads. Without the FIT framework, if we wanted to test this, executing "n" parallel transactions and expecting them to cause data corruption would be a vague way to automate a test. Even if data corruption occurs, it would be difficult to understand the cause of corruption. Each transaction would have multiple operations, so pointing out what operation in a transaction collided with another operation in another transaction would be almost impossible.

Resolution: We designed a latching mechanism that can be used to execute one transaction until it reaches the required failure point and then pauses the signal in the other transaction to reach the required failure point. Once the second transaction reaches the failure point, it notifies the first transaction to proceed. This is effectively what happens even for parallel transactions. So we came up with a set of collision points that could lead to write-write conflicts or dirty reads and used the latches to automate these test cases.

Please refer to the Java driver Code: driverParallelTransactions.

Examples of functionality tested/bugs found: Concurrent transactions were tested with this support.

Problem 5: Multiple Performers — Parallel Transactions for Different SDKs

Since we support transactions in multiple SDKs, the same logic can be used while testing concurrent transactions with different SDKs. For example, Java transactions vs. CXX transactions. In the above example, we connected to the same performer since we wanted to run parallel transactions for the same SDK. In this case, TXN A will connect to Performer A (suppose Performer A is using Java transactions) and TXN B will connect to Performer B (running CXX transactions).

Please refer to the Java driver Code: driverMultiplePerformers.

Examples of functionality tested/bugs found: Concurrent transactions with different SDK clients were tested with this support. Also helped us in ensuring transaction metadata is intact.

Conclusion

This architectural design of the FIT framework not only helped us in resolving our test automation challenges but also helped the transaction development in test-driven development (TDD) mode. 

Efficient test automation: Splitting the framework into a single driver and multiple performers helped us to develop parts of the framework independently. The developer of each SDK provided us with the performer and the QE could focus on the test automation, i.e driver. The developers could also add Unit tests into the driver so that all the tests for transactions are handled by this single framework.

Test-driven development (TDD): We have developed the Java performer and written all the tests needed to sign off the initial few versions of transactions for Java SDK. Once Java SDK was released and the development of other transaction SDK, i.e CXX and .NET  started, our development team had to develop the performer application while reusing the same driver application. This helped them in developing their SDK in a TDD fashion.

We hope you’ve enjoyed this article. We are adding more features to this framework and will be coming up with a new blog describing the new problems and new solutions. In the meantime, to learn more about the FIT framework, please contact me. To learn more about Couchbase transactions, please visit Couchbase Transactions