Functional API Testing With Postman, Newman, and BlazeMeter

Postman is one of the most popular and comprehensive tools for API developers and testers. We've previously covered Getting Started with Postman on this blog and explained the basic functionality of making requests and organizing them into collections. This time we'll dive deeper into using Postman for automated API testing and introduce a workflow that encompasses Postman and its command-line companion, Newman, for local testing and BlazeMeter's Functional API testing feature to run tests in a cloud environment.

Postman Collections vs. Swagger/OpenAPI

It's possible to export Postman Collections and then use API Transformer to convert them into the OpenAPI format so you can use them with Swagger and other OpenAPI tools.

OpenAPI, however, is a limited format when it comes to testing. An API definition can include possible requests and schema definitions for responses, so it's feasible to validate live API requests against them. It's not possible, however, to write custom assertions with code.

Postman, on the other hand, allows developers to write test cases in Javascript that don't look much different from traditional unit tests. It's possible to write tests for each request and even for the whole Collection. These test cases and their assertions run whenever the user sends a request in the Postman UI.

The Postman Collection Runner can be used to kick off all requests in a collection and run their assertions. It's also possible to export collections, including the complete testing code, and run them from the command line with Newman. Thanks to this capability, testing can be included in a Continuous Integration (CI) workflow. And, finally, BlazeMeter's Functional API testing feature has native support for Postman/Newman Collections so you can run tests from various cloud server locations in the BlazeMeter network.

Writing Tests in Postman

Let's walk through the entire process from the beginning. Our starting point is the Collection from the previous article on Postman, which included a sole method - Get Single Post - from the JSON Placeholder API. If you don't have this Collection, you can go back to the other article and follow the steps to create it or adapt the following instructions to a different Collection you have created.

Launch Postman and open the Collection and the request to display it in the main Postman window. Under the URL bar, you'll notice some tabs with request options: Authorization, Headers, Body, Pre-request Script, and Tests. Once you click on Tests, you'll see a source code editor in which you can type your test code and assertions.

The test environment is a sandboxed Javascript engine. Most Postman and test-specific functionality is available through the global pm object. The Postman developers have added a few helper libraries, which you can find in the Postman Sandbox API reference. Their test framework is based on ChaiJS, which supports different styles for writing test assertions. Postman recommends the behavior-driven-development (BDD) coding style.

You can add a test case by calling pm.test() with two parameters. The first parameter is a description of the test which will appear in logs next to the status indicator. The second parameter is a callback function which contains the actual statements for the test case.

Let's define a first test case that checks whether the server has returned a successful response with a 2xx status code:

pm.test("Request was successful", function() {
pm.response.to.be.success;
});

This code demonstrates the to.be.success shortcode that Postman adds to the response object. The syntax looks weird at first if you haven't used BDD before but it's easy to understand even for non-developers because it's written almost like a typical English sentence, except for the dots.

Click on the Send button now. After the server returned its response, have a look at the tabs in the lower section of the Postman window: Body, Cookies, Headers, and Test Results. Behind the word Test Results, you can see the number of tests executed and how many of them were successful. Those numbers are blue when everything was successful and red if any of the assertions have failed. Right now it should show a blue 1/1.

We have asserted that the server has returned its OK, but we're not checking the response. Does it contain all the fields we expect? Let's find out and write a second test case. Type or copy the following lines below your existing code:

pm.test("Response contains expected fields (userId, id, title, body)", function() {
let json = pm.response.json();
pm.expect(json).to.have.property('userId');
pm.expect(json).to.have.property('id');
pm.expect(json).to.have.property('title');
pm.expect(json).to.have.property('bady');
});

What does it do? It parses the response as JSON and then checks for the presence of some properties. With pm.expect() you can write all types of assertions using the BDD Expect style from ChaiJS, so feel encouraged to experiment.

Click Send again. You'll notice 1/2 lights up in red after Test Results. When you click on the tab, you'll see a description of what went wrong. Well, posts have a "body", not a "bady." Fix your test case and run again, this time it should show 2/2.

Congratulations, you have written your first test case with Postman!

Using Postman Collection Runner

The Postman Collection Runner provides an alternative user interface to testing and allows you to run all requests in a Collection at once. To try it, click on the Runner button in the main Postman window.

In the Runner window, select your collection and click on the big blue Run button below. Your tests will run, and you'll get the results to your assertions displayed.

Exporting Collections and Using Newman

For testing your Collection outside of Postman, you need to export it. Here's how to do that:

1. Close the Runner window if it's still open and go back to the main Postman window.
2. Right-click on your collection and choose Export.
3. Confirm using the latest Collection format and click Export again.
4. Choose a directory to store the file, change the name if you want, and save it.

Newman does what the Collection Runner does; it's just a different interface and way to interact. Installing Postman does not automatically install Newman, but you can get Newman from NPM. If you are not comfortable using the command line or NPM, though, you can skip the following instructions and move over to cloud-based testing in the next section. Otherwise, for the sake of completeness, let's examine how to do this as well:

1. Open a terminal or console window.
2. If Newman is not available on your system, type npm install -g newman to download it (depending on your system you may have to prefix the command with sudo).
3. Using the cd command, navigate to the directory in which you've saved the file.
4. Type newman run, followed by the name of your exported file.
5. See the results of the test on your console.

Testing Collections With BlazeMeter

It's time to move your Collection to the cloud and let BlazeMeter run your test cases and assertions. Here are the step-by-step instructions to accomplish this:

1. Log in to BlazeMeter, or create an account if you don't already have one.
2. Click Create Test, then choose API Functional Test.
3. The option to import existing test cases hides behind the three dots on the lower left section of the scenario (see screenshot below). Click them and choose Upload existing test script.
4. You'll see the Collection as JSON data. There might be a warning for config files which you can ignore since the Collection doesn't need any additional configuration. Click the big plus sign and then upload a file.
5. Choose Postman (Newman) under Test type.
6. Choose a location from where you want BlazeMeter to run the tests.
7. Click Run Test.

Once the test has completed, you'll see your results in the browser. Feel free to dig into the test results, where you'll find the two assertions you've created in Postman. Awesome!

Now that you've deployed your test cases, you can run them anytime through BlazeMeter's dashboard.