A recent Forrester Wave study has validated the test automation pyramid, concluding that API testing should be increased at the expense of GUI testing:
We define API testing as a method where the business logic is directly tested without execution of the GUI. However, with the exception of some special cases, the traditional solution requires a considerable amount of coding. It’s an extension of unit testing, where the business units are integrated. In this case, reliable testing knowledge is also required. Furthermore, this is not an agile method, so many teams continue to test through the GUI.
I think that the cause of the current small ratio of API testing is the need for reliable testing knowledge, as well as the considerable coding work needed. The maintenance cost is also significant. In order to increase the use of API testing, a different method is needed, one which eliminates the current shortcomings of API testing.
Keyword-driven testing is a software testing methodology which separates test design from test development. Among other benefits, this method allows additional professional groups such as business analysts to work on the test automation process. By applying KDT, test cases can be created without implementation issues and independently from the writing of test code. In this way, functional tests can be developed prior to any implementation work.
Our new Keyword-Driven API Testing or KDAT is a test-first agile method where the Graphical User Interface is replaced by a Test Tool Interface (TTI). A TTI is a piece of code that can send input to and receive output from the business logic. A GUI is a connection between the person and the business logic, while the TTI is a connection between the test tools and business logic. During GUI test automation the automation tool sends and receives data to and from the GUI. During KDAT all data are transferred along the Test Tool Interface:
The method signatures of the Test Tool Interface can be marked out as keywords. The test tool is able to recognize and call these methods (keywords). In this way, the necessary input for the business logic can be set by the test tool as keyword parameters, and no GUI is executed during the testing process.
TTI can be created at any phase of the development or the maintenance, however, the best solution is to plan this interface as part of the software in the design phase. In this way little additional coding is necessary.
The process of KDAT is the following. The agile team, just like the Three Amigos, make the test design. The result of the test design is a test table with keywords and parameters, as detailed here. Based on the keywords, the TTI code part is planned together with the “traditional” code. The test table can be automatically converted to a test file, which is then understandable for the testing tool. When the code is ready, tests are executed and the test results are evaluated.
In some cases, it is well worth applying KDAT to an existing system. Crucial or often-changing parts of the code can be the subject of the late automation. In this case making the TTI is more difficult, but manageable. Our recent case study, where the application under test was Monopoly, revealed that the total test automation cost of KDAT and GUI testing was approximately the same. In this case study, keywords and test cases created manual, though our 4Test method enables the generating of executable tests for KDAT. Since the system is already working after the creation of the test table, the tests can be executed.
It’s reasonable to consider advantages with respect to GUI testing and existing API testing separately. Considering traditional API testing, the advantage of KDAT is that testing and coding work is separated and everybody is able to focus on what they do best. With the keyword-driven method, KDAT can be used in agile projects and the tests will be understandable for everybody. Another significant advantage is that much less code to be written. In addition, this code is more stable since the tests are changing more frequently than the keywords, based on the test developed.
On the other hand, API tests are much faster than GUI tests, meaning the former can be executed several times a day. Obviously, the tests are also more stable, since they are immune to GUI changes. In addition, KDAT is cheaper than GUI testing in the maintenance phase because of the stability, and for greenfield projects, it’s cheaper for both the development and the maintenance phases.