HTTP API: Key Skills for Smooth Integration and Operation (Part 1)

I've been a web developer for quite a few years now, and during my career, I've encountered a significant number of various projects — both large and small. In these projects, I've had to solve a wide range of issues. In this article, I want to share my experience on how to work properly with HTTP APIs of third-party or internal services.

Throughout their work process, developers face various problems, from structuring code to setting up databases. Integrating third-party APIs is a particularly complex task, where unforeseen difficulties can arise, often affecting the company's revenue and reputation. I know this firsthand since years ago, I developed systems where the lack of properly established integration cost money and the reputation of the company I was working for. That's why it's critically important to learn how to solve problems associated with APIs, especially considering their unpredictability and the possibility of sudden changes, even from the developers' side: sometimes they intentionally “break” the API or set unexpected limits, and there may be no documentation about these features whatsoever.

With the advancement of technology, the ability to integrate various services through APIs is becoming increasingly demanded. Not so long ago, we all witnessed the rapid growth in popularity of the software development approach that involves breaking down monolithic systems into services or microservices. Many of these microservices interact with each other precisely through APIs.

The importance of skills in integrating various services through APIs is evident in the growth in the number of startups based on API interaction. Furthermore, Postman's 2023 study notes the popularity of HTTP APIs, including REST and GraphQL.

My narrative will focus on REST, but the proposed solutions may also apply to other architectures. This material is divided into two parts: the first will examine the main problems that arise when integrating services through APIs and discuss the use of queues to overcome them; the second will be dedicated to working with data.

Let's get to the heart of the matter.

What Could Go Wrong?

Let's say you've developed the backend of a system, and you're faced with the task of loading information (such as profile pictures) from an external service via API to complete user registration. You decide to send the request directly and synchronously, without complicating the process with unnecessary optimization. At first, everything seems to be working fine; registration goes through a bit slower, but without any visible reasons for concern.

However, problems often strike unexpectedly, and the system starts to malfunction when requests begin to fail due to timeouts, causing the application to hang and blocking the registration of new users. As a result of the failure in the operation of the external service, which was not anticipated in advance, the company faces the loss of customers and financial losses, necessitating immediate action to resolve the issue.

Now, Straight to the Problems

There are numerous scenarios where API failures can lead to serious issues in the operation of a website or application, or affect their individual components. Here are some of the main ones:

1. API availability errors/server errors: An example was mentioned earlier. In the context of HTTP APIs, typical errors usually have codes starting with 5. However, it's important to check the API documentation, as codes might be used non-standardly.
2. Rate limiting: APIs may have limits on the number of requests to reduce load or as part of paid services.
3. Strange or unexpected responses from external services: APIs might return non-standard responses, for example, HTML instead of JSON due to blocks. There are cases when DDoS protection can change the content of the response by adding HTML with a captcha. Therefore, you need to be prepared for different outcomes.
4. Security issues: Security is critical, but API users often forget about it. At first glance, it might seem that sending a request to an external service is safe and cannot lead to vulnerabilities or financial losses in your project. However, the risks vary: from disclosing the server's IP address to XSS attacks.

I've only listed some of the possible problems, but there are quite simple rules to follow to save yourself from such errors, as well as those I haven't included here. Let's try to figure it out.

Queues for Sending API Requests

There can be both synchronous and asynchronous requests. If the application of synchronous requests is done, a heavier reliance on external services will be an evident problem, which can, however, be dealt with through the adoption of asynchronous requests. This approach covers some alternatives for setting up a system to send out messages asynchronously, and one of the most preferable ones is using queues. Asynchronous submission via queues not only reduces dependency on third-party services but also improves control over requests. It brings more optimization, like being able to repeat or prioritize the course, thus helping the speed and reliability of working with APIs better.

Nevertheless, harnessing the power of queues to improve API interactions is not always free of pitfalls. It's crucial to understand the nuances of a queue system necessary to reap the maximum rewards from its use. Specifically, one must comprehend how queue optimization techniques can be applied effectively. Let’s consider them.

Queue Optimization

Exponential Backoff

When using queues for API requests, it's important to consider that an excessive number of tasks can overload both your system and the external service. This happens because each request necessitates a certain amount of computational resources, and overload can cause performance degradation in both systems. Therefore, balancing the frequency and number of requests is crucial to prevent such issues.

One solution is to use the method of exponential backoff for each subsequent API call: if a request fails, the next attempt is made after a doubling period — first after 1 second, then 2, 4, and so on. This prevents the accumulation of requests and enhances the system's resilience. It's also necessary to set a maximum retry interval and implement monitoring with alerts for quick response to problems.

NB: do not make repeated requests for errors with codes starting with 4 without taking additional corrective actions, as error codes starting with 4 usually indicate that the request itself was sent incorrectly (these are client errors).

Grouping Requests and Preventing Race Conditions

When sending requests asynchronously, especially through a queue, a race condition can occur when the simultaneous processing of similar requests disrupts the order of data on the external server. 

There are several solutions to the problem. For example, a simple solution is to block requests of a certain type until all previous ones are completed. Blocking can be done using a unique entity identifier or the URL of the request – this solution will help eliminate race conditions. The specific implementation of the block depends on your system type and how queues are processed, whether on a single server or across multiple servers. 

The topic of request blocking in distributed systems is quite complex; nevertheless, the issue can be resolved by maintaining the state in a database and using the SELECT FOR UPDATE query to change this state, or by using systems like ZooKeeper to create a distributed locking service. How you implement this functionality will greatly depend on the solutions you've used in earlier stages and the overall complexity of your system.  

Another efficient approach would be to group requests by entities, processing only changes from the last saved state. This method allows for the elimination of tasks that return the entity to its original state and the merging of fields or filters for these requests, optimizing processing and avoiding repetitions.

Moreover, by using request grouping in tandem with solving the race condition problem, you can also reduce the number of requests, which is crucial when dealing with limitations on them. And if the API is paid, then costs will be reduced.

What also makes a lot of sense is combining both described solutions. This combination can help in scenarios where a grouped request has already been sent, the queue handler has not yet received a response, and a new request arrives. This approach ensures efficient and orderly processing of requests, significantly reducing potential issues related to race conditions and request limitations.

Pre-Calculating Maximum Request Timeouts

Beyond the mentioned queue optimization methods, it's necessary to estimate the maximum time for sending each request, considering the server's limited resources and queue handlers. Delays can occur due to network issues or problems on the remote API server's side. It's important to configure values for connection timeouts and request execution timeouts.

Determining the optimal timeout requires considering numerous factors: typically, the connection timeout is set shorter than the request timeout, as establishing a connection takes less time than processing it. The optimal values depend on the location of the API provider's servers relative to your own.

For precise adjustment, it's essential to collect API performance statistics, for example, in shadow mode, which allows analyzing performance under conditions as close to real as possible. Data can be gathered using tools like curl (with the --write-out option) or libcurl methods (curl_easy_getinfo to obtain CURLINFO_CONNECT_TIME and CURLINFO_STARTTRANSFER_TIME).

Based on the collected statistics and data analysis, you can establish permissible timeout thresholds for efficient API interaction.

Maintaining HTTP Connections

We've already discussed the importance of careful timeout management when sending requests and their significance in the context of queues. Now, it's worth highlighting the use of keep-alive in HTTP servers as an effective method. It allows for the reuse of TCP connections for multiple requests, reducing server load and speeding up processing. Most HTTP clients, including curl, support this feature.

It's important to be mindful of the API servers' limitations on the number and duration of connections. In cases of restrictions, Domain sharding can be useful, but using HTTP/2 is preferable due to its efficiency and the ability to multiplex requests over a single connection. This improves performance by allowing simultaneous request sending and receiving. Modern tools, such as curl, default to using HTTP/2 (when possible) to optimize data transmission.

Batching/Advanced Filtering

Another optimization method is batch processing. This involves sending groups of data in one request to reduce the number of requests. Tasks are grouped and processed at intervals, for example, every 5 seconds, optimizing overall performance. It's important to control the volume of data, especially under high load, by adjusting the sending intervals. This approach works for both creating or modifying entities and requesting them through filtering, allowing efficient data retrieval based on necessary criteria.

Request Prioritization

Not all API requests are equal: critical tasks, such as financial transactions, should take precedence over less important ones, like image uploads. Priority queues ensure that important requests are processed first, even if they arrive later. It's crucial to properly configure the priority system to prevent delays in processing less critical tasks due to a constant influx of more important ones. Implementing adaptive priority adjustment can help the system efficiently adapt to changing loads. It's also recommended to separate the processing of different types of requests using specialized mechanisms to enhance overall performance.

In this part, we've explored key ways to prevent issues with API usage through queue optimization, focusing on methods such as exponential backoff to avoid system overload, grouping requests to prevent race conditions, and the importance of managing timeouts, as well as highlighting the need for adaptive request priority management for efficient application operation and the use of keep-alive connections to reduce server load. In the next part, we'll discuss leveraging caching and employing data, and wrap up our insights.