Best Practices for Developing API Connectors for Clouds

Cloud service providers like Amazon Web Services, Microsoft Azure, Google Compute Engine, etc. and their platforms like OpenStack, vCloud, OnApp, etc. are increasingly programmable through APIs or web services. To consume these APIs/web services, we need to develop a connector. We have experience developing API connectors to almost all the cloud platforms or services out there today. This blog is to share our experiences and provide some of the best practices to be followed for developing web service connectors to cloud services or platforms

Why API Connectors

API connectors are required for the following reasons:

1. Discover cloud resources
2. Manage resources
3. Automate operations
4. Orchestrate resources
5. Integrate with tools/products
6. Report summaries of resources and their utilization

Steps for Developing API Connectors

The following are the sequence of steps to develop efficient and effective connectors to cloud platforms/services

1. Understand API characteristics
2. Analyze API support provided by cloud platform/services
3. Identify API operations
4. Validate API endpoints
5. Verify API authorization
6. Check quota
7. Analyze cloud resource pricing
8. Design considerations for the API connector

API Characteristics

The API Characteristics includes API Type, Authentication mechanism and Request/Response type.

API Type

Typically, APIs are exposed either through REST (Representational State Transfer) or SOAP (Simple Object Access Protocol). REST is becoming the standard and replacing some of the old SOAP APIs. This is very evident as per the data in Table 1, which is farther down in the article.

API Authentication

Every cloud platform uses different types of authentication mechanisms to access the APIs, and it is important to understand those authentication mechanisms. Typical examples are:

• Basic authentication
• Token-based authentication
• SSL authentication
• Multi-factor authentication

Basic Authentication

Basic authentication uses the classic combination of username and password encoded in base64, which is supplied in the authorization HTTP header.

Example: Authorization:Base64(username:password)

Request:

Headers: Content-type: application/xml
Authorization: Basic dG9ib3RyYXM6cTE=
Method: GET
API: http://api.xyzcloud.com:4465/paci/v1.0/ve
IPaddress | host - api.xyzcloud.com
PortNo – 4465
Path – paci/v1.0/ve

Response:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> 
<ve-list>
    <ve-info subscription-id='100001' name='web' state='created' description='Web Server'/>
</ve-list>

Token-Based Authentication (X-Auth-Token, Oauth Token)

Token-based authentication provides temporary token based on user credentials for authentication. A resource needs to refresh temporary tokens when they expire. An internal authentication handler authenticates based on the provided tokens in the header.
Request:

Headers: Content-type : application/json
Method: POST
API: https://api.myapplication.com/user/accessToken

Body:

{
    userName:'my username',
    password:'my password',
    token_type:'mac'
}

Response:

{
    “access_token”: “okKXxMWOEhnM78Rie02ZjWjP7eQqpp6V”,
    “mac_key”:”vV6xEfVgQZv4ABJ6VZDHlQfCaqKgFZuN”,
    “mac_algorithm”:”hmac-sha-1”,
    “token-type”:”mac”,
    “expires_in”:3600,
    “refresh_token”:”nZSiH3L5K4febMlELguILucrWpjRud56”
}

SSL Authentication

The Secure Sockets Layer is a standard security technology for establishing an encrypted link between a server and a client – typically a web server (website) and a browser. SSL authentication requires an SSL certificate to be uploaded to the platform under the subscription. The API endpoint needs to be authenticated via SSL certificate.

Multi-Factor Authentication

Multi-factor authentication (MFA) adds an extra layer of protection on top of a username and password. MFA-supported APIs require a username and password for the first factor and an authentication code from an MFA device as the second factor. These multiple factors provide increased security for API endpoints.

API Response Type

The request and response type of the API needs to be looked at for feeding inputs and consuming the output. The API request and responses are either in XML or JSON. The connector needs to convert the response as per its interface requirement

API Support

API connectors for cloud platforms/services can be developed by the following options

• Consume REST/SOAP API directly using the programming language of your choice like Python, Java, .NET, Ruby, GO, Node.JS, etc.
• Some cloud platforms/services offer SDKs that wrap around the API to make it easy for developers to consume programmable SDKs specific to Python, Java, .NET, Ruby, etc. provided by the platforms/services
• Use third-party libraries such as LibCloud (Python), Fog (Ruby), jCloud (Java), Libretto (JavaScript), pkgcloud (Node.JS), or gophercloud (GO) based on their support.

The table below shows API support for some of the leading cloud service providers and platforms.

Note: These Cloud service API details are available in Google Sheets. Encourage service providers and other readers to update their API endpoints for the cloud so that it acts as an easy reference for developers.

API Operations

Understand the API operations supported by your platform by going through the API documentation and identifying the operations that you would like to consume. It is better to perform the operations through the management portal or dashboard to get a sense of how it works before you start consuming it through the API. The first thing that you need to do to consume the API is authenticate, and then you can try basic read operations before performing create options.

Validate API Endpoints

API endpoints are not the same as a cloud platform management URL. An API endpoint typically includes host, port, and path. It includes an access key and a secret key if it is a REST API. Validate the accessibility of API endpoints for the platforms or services using tools like POSTMAN, RESTClient, etc. For token-based authentication, we need to generate the token and feed the token in RESTClient.

API Authorization

After the API is authenticated, we need to know the authorization for a given user in the cloud platform or service. For example, with AWS Identity and Access Management (IAM), we may have successfully authenticated, but we can perform only the action that we are authorized in the IAM.

Quota

Cloud platforms/services impose quotas for resources that can be consumed by user accounts. It is better to understand the quota limits upfront. For example, say AWS limits the allocation of elastic IP to 5 for an account. However, this can be increased by raising a request. OpenStack admins can define limits to resources in each project consumed by the user.

Analyze Cloud Resource Pricing

It is very important to check resource pricing by a cloud service provider. Cloud service providers charge resources on a monthly, hourly, or per-minute basis. It is important to know the pricing before consumption, otherwise, we will have surprises in our billing. It is also important to understand the free tier offered by service providers in detail so that we don’t have any surprises.

Design Considerations for API Connector Development

• It is ideal to use SDKs provided by the platform if you are developing a connector to only one platform.
• If the platform does not provide SDK for the required language, there are tools like APIMatic, AWS API Gateway, etc. which helps in generating SDKs for the API Endpoint. The use of SDKs in connector development reduces development effort.
• If you are looking to develop connectors across multiple clouds, you can consider using third-party SDKs, as it helps in accelerating development. However, if you want the connector to be dynamic and you need to evolve with the platform or services, it is better to go with the SDK provided by platform or services, as third-party SDK support for some of the new releases may take time.
• Understand the API rate limit (the number of API requests that can be made to the API endpoint in a period by a user) set by some of the providers and platforms, as it gives an indication of how frequently we can make calls to an endpoint.
• For some asynchronous APIs (where the API response is not immediate), the responses are provided through PUSH or retrieved through POLL. The ‘Push’ model expects a callback endpoint to which it sends the response when it is available. In ‘POLL’ model, the requester calls an API repeatedly to check for an update in status. When you have to poll or retry an API request, we recommend using an exponential backoff algorithm to calculate the sleep interval between API calls. The idea behind exponential backoff is to use progressively longer waits between retries for consecutive error responses.
• Some cloud service providers/platforms expose different endpoints for each service to be consumed. It is recommended to maintain a service catalog with API endpoints to ensure you're using the right ones.
• Sometimes the endpoint varies based on the sub accounts of the cloud platform or services. Ensure that the endpoints are concatenated as per the requirements before making a call

Hope this helps. Have fun developing API connectors…