The Modern Way of Managing APIs Using Protobuf and OpenAPI

I am a tech-enthusiast, and I love sharing new technologies, ideas, and innovations about software development. This time, I will be talking about API contract management with Protobuf and OpenAPI.

One of the most important aspects of microservices-based applications is the ability to deploy microservices completely independent of one another. To achieve this independence, each microservice must provide a versioned, well-defined contract to its clients. Each service must not break contracts until it's known that no other microservice relies on a particular contract version.

In this post, I am going to present a project containing patterns, tools, and technologies used to develop contract-based APIs which will help teams communicate and ensure compatibility.

Bootstrapping a Modern API

The idea of this project is to provide you with a bootstrap for your next API based project(e.g microservices). We are addressing some of the main challenges that everyone faces when starting with this type of project. This project will definitely help you get an understanding of the API lifecycle and save you a lot of time when setting up your initial API architecture.

In this blog post, I will walk through some concepts and technologies that we have placed in our bootstrap modern API management project. I’m not planning on going very deep into the concepts and tools — we have a lot of posts about those out there — the intention here is to present an application example containing everything when developing contract-based APIs.

This project tries to provide a modern approach to manage APIs, combining the Protobuf and OpenAPI specifications.

 The project tries to solve the following challenges:

• Contract definitions
• Contract compatibility enforcement 
• Contract versioning
• OpenAPI definition generation
• Stub generation to different languages (Java and Golang)
• Client SDK generation to different languages(Java and Golang)
• Web UI for manual test
• API interactive documentation
• API management guideline 
• ETC

API Contract

Is a definition that describes the surface area of the request and response of each individual API method being offered. It is something that both API provider and API consumer can agree upon, and get to work developing and delivering, and then integrating and consuming. An API contract is a shared understanding of what the capabilities of a digital interface are, allowing for applications to be programmed on top of. An API contract should be provided ahead of time for of each version, allowing consumers to review that new version of an API contract before they ever commit to integrating and moving to the next version. 

Google Protocol Buffers: Protobuf

In our project, we leverage from a binary protocol such as gRPC and extend the use of Protobuf(IDL) for external-facing APIs based on HTTP protocol. 

From the Google Protocol Buffer website:

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages…

The .proto file is the source of truth for our service design. The first step when designing our service is to write a .proto file then we use it to discuss the design with who will consume the API before starting the implementation.

From our proto definition, we will generate the OpenAPI definition file which will be the source to generate interactive documentation, SDK clients to our HTTP API version, and web client to manual testing.

Code Generation

Everything starts with the Protobuf and from there boilerplate code, documentation, definition file, stub, and SDK clients are automatically generated. Inside the bin folder you will find a bunch of scripts to generate everything you need:

• Java and Golang gRPC stubs
• Golang generated code deployment
• Java packaging and deploy
• OpenAPI definitions
• Java and Golang SDK clients
• Swagger HTTP client UI 

API Compatibility and Design Best Practices

Protobuf APIs should be stable so as not to break consumers across repositories. To ensure back-compatibility, we use the Buf CLI tool. In the same way, we are using Buf to run lint rules to help you write clean, consistent, and maintainable APIs. Both checks are done every time a new Pull Request is opened, and you can look at the jobs configuration in the Github actions

RPC vs. REST

RPC-based APIs are great for actions. REST-based APIs are great for modeling your domain, making CRUD available for all of your data. However, We’re not simply storing stuff on the web; no, we are doing stuff(Actions). Most of the systems are complex. Don’t try to shove your creativity into the world of CRUD. REST for microservices is a terrible idea.

Unfortunately, REST becomes a marketing buzzword, and most systems saying they’re REST are little more than RPC with HTTP verbs and pretty URLs. These APIs are jokingly called REST_ish_ by people aware of The REST 4 levels of maturity. 

For internal systems, RPC communication is becoming the standard thanks to new technologies like gRPC, but most developers are still using REST for external-facing services. You should check out why Slack uses an RPC-based Web API, and why their API would not fit into REST nicely, you may find out you should be using RPC for your HTTP services as well.  

In our project, we recommend using RPC for internal systems and suggest considering RPC also for public APIs.

 GraphQL can be an alternative for the HTTP-based REST APIs but it is out of scope for this article.

gRPC vs. HTTP RPC

We prefer to use gRPC for internal APIs for many reasons but you might choose to add extra support for RPC over HTTP. It is totally fine, especially for the API gateway. Browsers don’t support gRPC natively and having a web client to test an API can be handy.

In our project, we show how easily add HTTP support to your gRPC service without using any library. I strongly discourage using grpc-web and grpc-gateway solutions, those approaches add unnecessary complexity to your service.

Guideline 

As we work with APIs we gain experience and we have shared some rules and best practices that might help save you time and headaches. Check it out.

Conclusion

If you are starting a new microservice architecture, you will certainly use gRPC with Protobuf. While you are working with these new technologies you will face many design and management challenges which will require some experience to make the right decisions. This project has gathered many tools, scripts, and patterns to help you save some time.