Building Robust Automation Scripts With REST APIs

From Native to REST API

The availability of REST APIs made the task of integrating systems easier. Earlier, only native APIs, application CLIs, and various SQL interfaces such as ODBC were available. So, integration projects had been specific to the applications involved, except for the SQL-based interfaces that brought some amount of uniformity with ANSI SQL and ODBC API calls.

The knowledge of how to use REST APIs is an important requirement for any DevOps engineer these days, mainly due to the following reasons:

• A typical automation project would require integrating applications. Making calls to REST APIs is common even in some glue code.

• Even if no integration work is needed, to build useful tools that query information from an application such as a monitoring or business application, you still may have to use REST APIs.

The Unreliable HTTP(S) Call

A REST API call is made over HTTP or HTTPS protocol and very often over the Internet. A single HTTP call is not very reliable because there are no built-in retry options. Even if the underlying application that serves the API is available, an HTTP call could fail due to external issues such as network latency or temporary unavailability of Internet in the case of accessing a SaaS application.

One common method to mitigate the above problem of unavailable network or application resources involved in a REST API call is to introduce retry in the calling scripts. Even though there are several ways to implement it, the simple tutorial provided here illustrates how retry can be implemented in PHP code using the curl library when making calls to REST API.

The logic used here could be implemented in any scripting or programming language, with or without curl support.

The PHP/Curl Tutorial

PHP supports the curl API if the libcurl package is installed on the machine where the script runs and PHP is installed on it compiled with the curl option.

Curl Wrapper Class

The curl wrapper class provided here makes our life easier when calling REST APIs from scripts. You can use curl functions directly in your PHP code, but the code would be verbose with all the error handling and status checks that would be added with each call, which can overwhelm someone reading the code (including you when you come back to your code later to change or fix something).

The wrapper class has all the error handling, and, most importantly, the retry implementation that would make the scripts robust.

CurlWrap.php:

<?php

   define('CURL_EXEC_RETRY_MAX',3);

   class CurlWrap
   {
      private $ch;
      private $url;
      private $response;
      private $info;
      private $http_code;

      function __construct()
      {
         $this->ch = curl_init();
         $this->setOption(CURLOPT_RETURNTRANSFER,true);
         $this->setOption(CURLOPT_BINARYTRANSFER,true);
         $this->setOption(CURLOPT_FOLLOWLOCATION,true);
      }

      function __destruct() {curl_close($this->ch); }

      public function setOption($option,$value) {curl_setopt($this->ch,$option,$value);}

      public function exec($url='')
      {
         if ($url!='') $this->setOption(CURLOPT_URL,$url);
         $this->retryExec();
         $this->info=curl_getinfo($this->ch);
         $this->http_code=$this->info['http_code'];
      }

      public function getHttpCode() {return $this->http_code;}
      public function getExecResponse() {return $this->response;}
      public function getExecInfo() {return $this->info;}
      public function getError() {return curl_error($this->ch);}

      //The logic of retry can be different, but when making a web service call
      //it is essential have some retry, as the resource might not be accessible briefly online
      //due to many reasons.
      private function retryExec()
      {
         $i=0;
         while ($i++ <CURL_EXEC_RETRY_MAX) {
            $this->response=curl_exec($this->ch);
            if ($this->response) break;
            if ($i<CURL_EXEC_RETRY_MAX) sleep($i);
         }
      }

   }
?>

Using the CurlWrapper Class

Using this class to make REST API calls from a PHP script is easy. Look at the following sample program, which fetches the RSS feed for the DZone home page.

test_curlwrap.php:

<?php

include_once 'CurlWrap.php';

$curl=new CurlWrap();
$curl->exec("http://feeds.dzone.com/home");
if ($curl->getHttpCode()!='200')  print "ERROR: Failed to fetch url. ". $curl->getError(). "\n";
else print $curl->getExecResponse();

?>

This will just print out the RSS feed returned by the URL.

Now, try changing the domain to something non-existent, like http://www.dzonexyz.com/. Then, you will see an error message like this:

ERROR: Failed to fetch url. Couldn’t resolve host ‘www.dzonexyz.com’

The default HTTP request method used by curl is GET, and that’s what we used in the last example. REST APIs typically use GET for reads, POST for creating a new entry, and PUT for updating an existing entry. In such cases, related options must be set prior to making a call to the API.

The Takeaway

The main point of this article is to stress the requirement of making automation scripts robust. Failure of automation scripts can play havoc with production systems which would undermine the credibility of DevOps group.

The REST API calls from scripts can be made more reliable by adding simple retry logic as illustrated in the example. It is appalling to see that sometimes even application code would lack such basic precautions and error handling that would help to avoid mishaps in production environments. Hoping for the best is not a viable strategy in automation.