API Management vs API Gateway: Where Does API Analytics and Monitoring Fit?

For the last few years, there has been an explosion of API-powered businesses. There are revenue-generating APIs, developer platforms, partner marketplaces, and even internal APIs powering single-page apps.

With this explosion, there has also been a large increase in API tooling to help these companies go to market with their API platforms as quickly as possible and out-innovate any competition. Much of this increase in tooling mirrors what we saw in the mobile era. However, with this explosion, there is now an increase in the number of tools and solutions to build and grow APIs and platforms.

In addition, tools like API gateways can be used both as a proxy in front of public-facing APIs but can also behave like a service mesh orchestrating between various internal services. This guide aims to provide an overview of various API tools.

API Gateways

An API Gateway is an HTTP server/proxy server that sits in front of your API and provides a central place to administrate, route, and secure your APIs and services. API gateways should be deployed in clusters so you can scale the system horizontally by adding more machines and ensure high availability. 

API gateways are usually on-premise appliances, but modern ones are usually based on an open-source or open-core model. For example, Kong is based on NGINX and Express Gateway is based on Node.js and Express.

There are also proprietary, cloud-based solutions from cloud vendors, like AWS API Gateway and Azure API Management. Usually, an API gateway will do the following:

Authentication and Rate Limiting

A primary objective of deploying an API gateway is to provide a secure way to access your APIs and prevent malicious activity. If your API requires authentication, an API gateway provides an easy way to generate and also manage API keys for each consumer. Clients accessing your API without the correct credentials will get a 401 Unauthorized. 

To ensure fairness, security policies, such as rate-limiting can be enforced, such as a limit of 100 requests per minute per API key. Rate limiting can be enforced based on API keys, via an IP address if no authentication layer is added or other custom policy. You can even include additional security policies, such as bot detection and prevent the HTTP payload from being too large.

Caching and CORS Policies

Caching enables your API platform to handle a higher number of clients and absorb peak traffic. Certain types of content, such as e-commerce and travel can benefit greatly from caching. Others, such as banking and financial services, may not benefit from caching, so you need to perform a cost-benefit analysis whether caching makes sense for your applications. Other policies, such as CORS (Cross-Origin Resource Sharing) can be enforced to allow the API to be accessed from a web browser.

Data Validation and Transformation

An API gateway provides a central mechanism to add data transform rules to your API. Data transformation can be as simple as URL rewrite, such as api.example.com/search to search.example.com to more complicated transform rules, such as transforming XML to JSON. This could be handy if you already have many legacy internal services that you want to expose as an API, even if the legacy services use an older content format such as XML instead of JSON.

In addition to data transformation, some API gateways can also combine end to avoid API chaining. This enables your customers to leverage a single public endpoint, like GET /user/me, even though it has to fetch from various internal services like your authentication service, user service, and billing service. 

The responses from those internal services are combined into a single response to be returned to the client. This makes working with your API easier for customers since they don’t have to perform API chaining. An extreme example of this would be Apollo GraphQL, which can fetch many entities from many services and combine them into a single endpoint POST /graphql

API Canary Release/Versioning

Similar to data transformation, with an API gateway, you can leverage an API gateway to route users to specific versions of your services. This can be done via a header field, a path segment such as /v1 or /v2 or even handle sophisticated canary releases where a percentage of user traffic is routed to a new version. 

The ratio of users hitting the updated API version can slowly be increased as your confidence in the release increases. By performing a gradual ramp, you minimize the risk that a new API release becomes a system-wide outage due to functional or performance problems.

API Analytics

Once you invested in your API platform, you’ll want to ensure you can get a return on your investment by building an API platform that customers actually adopt and is bug-free. An API analytics solutions like Moesif complements your API gateway by providing you with the necessary visibility and reporting needed to create product lead growth. This includes how your customers are using your APIs, who they are, and which marketing channels lead them to integrate. In addition to reporting, API analytics usually includes real-time monitoring capabilities so you can stay alerted when issues do occur.

While API management and API gateway solutions are designed for infrastructure and site reliability engineering in mind, API analytics usually is designed with a broader audience of both technical and non-technical users including product, engineering, and support. Because of this, API analytics vendors are more likely to be managed SaaS solutions, rather than on-premises, but can be homegrown also. 

API Product Analytics

Product analytics enables you to track and understand key API metrics like your API DAU (Daily Active Users), the endpoints your top customers are using, and 90th percentile latency for key endpoints. Some API analytics tools provide more sophisticated analysis like cohort retention analysis and funnel analysis so you can track KPIs like average Time To First Hello World (TTFHW) and conversion rates so you can understand how engaging and sticky your product is.

TTFHW measures how long it takes from first visit to your landing page to an MVP integration that makes the first transaction through your API platform. This is a cross-functional metric tracking marketing, documentation and tutorials, to the API itself.

API Logging/Debugging

Besides measuring product metrics like retention and engagement, API analytics also provides an easy way to speed up investigating and debugging API functionality and performance issues. High-cardinality log analysis enables you to plot trends using heatmaps, time series, waterfall diagrams, and other charts. Some API analytics also log the full API calls in real-time so you can inspect the request and response headers and body and do a diff or even replay and generate tests from failing transactions.

API Monitoring

Many API analytics tools also provide API monitoring and alerting capabilities so you can get notified when an API is down or acting incorrectly. Unlike synthetic monitoring, which are simple health probes, API Real User Monitoring, detects anomalous behavior from your actual API traffic made by your customers based on historical trends. API issues could be business-related, such as an unexpected drop in API activity from key customers, or it could be engineering related such as functional or performance problems. Specifically, API analytics focuses on a few key pillars:

Team Dashboards

Some API analytics solutions provide a way to share and collaborate on dashboards with both technical and non-technical team members or automatically sandbox the usage data so you can share dashboards directly to your external partners in a secure way. By embedding an API debug log and plan usage information in your customer-facing portal, customers can resolve issues without overloading your already stretched support team.

User and/or Company personas

API analytics tools pull data from your CRM, BI, and support tools. This enables your customer success and support teams to look up a customer’s API activity by their email or name to resolve integration issues quickly. User and company level information also enables your company to centrally track how various prospects and customers are integrating and sticking with your APIs.

API Management

While API Gateways and API management can be used interchangeably, strictly speaking, an API gateway refers to the individual proxy server, while API management refers to the overall solution of managing APIs in production which includes a set of API gateways acting in a cluster, an administrative UI, and may even include additional items such as a developer portal for customers to sign up and generate new API keys.

The admin panel enables a few things:

• The ability to add and remove plugins such as for security
• A way to edit and visualize data transforms such as XML to JSON
• Upload and administrate SSL certificates