API Governance Models in the Public and Private Sectors: Part 5
Join us as the API Evangelist, Kin Lane, shares a detailed report concerning the U.S. Department of Veteran Affairs and its wish to understand API governance.
Join the DZone community and get the full member experience.Join For Free
This is part five (you can find part four here) of an eight-part series on the Department of Veterans Affairs microconsulting project, “Governance Models in Public and Private Sector.” Providing an overview of API governance to help the VA, “understand, with the intention to adopt, best practices from the private and public sector, specifically for prioritizing APIs to build, standards to which to build APIs, and making the APIs usable by external consumers.” Pulling together several years of research conducted by industry analyst API Evangelist, as well as phone interviews with API practitioners from large enterprise organizations who are implementing API governance on the ground across the public and private sector, conducted by Skylight Digital.
We’ve assembled this report to reflect the interview conversations we had with leaders from the space, helping provide a walkthrough of the types of roles and software architecture being employed to implement governance at large organizations. Then, we walk through governance as it pertains to identifying possible APIs, developing standards around the delivery of APIs, how organizations are moving APIs into production, as well as presenting them to their consumers. Wrapping up with an overview of formal API governance details, as well as an acknowledgment that most API governance is rarely ever a fully formed initiative at this point in time. Providing a narrative for API governance, with a wealth of bulleted elements that can be considered, and assembled in the service of helping govern the API efforts across any large enterprise.
Development to Production
After understanding the roles needed to realize governance, more about the underlying platform architecture that is needed, how organizations can identify where the API opportunities are, and making sure groups are putting standards to work, we scrutinized how groups are moving APIs from development to production in a more structured way. Governing how teams are efficiently moving APIs from idea and design, to actually putting services to work in a production environment at scale across large teams. Documenting the lifecycle of a service, and the common elements of how enterprises are getting the job done on a regular basis.
To be able to deliver APIs at scale in a consistent way, teams are relying on a well-honed, well-defined lifecycle that has been defined, proven, and evolved by senior teams. Forcing structure and rigor throughout the evolution of all services, putting governance in front of teams, and forcing them to execute in a consistent way if they expect to reach a production reality with their services. Focusing on a handful of structured formats for imposing governance at the deployment levels.
- Contract — Requiring ALL services to begin with a machine-readable OpenAPI contract defining the entire surface area of the API and its schema. Leveraging the contract as the central truth for what the service will deliver, and how governance will be measured throughout the lifecycle of the service.
- Process — Providing a well-defined process for all developers laying out how any service moves from design to production, with as much detail regarding each step along the way. Helping all developers understand what they will face as they work to move their services forward in the enterprise.
- Scorecard — Having a machine-readable checklist and scorecard with tooling to help each developer fork or establish an instance for their service. Providing a checklist of everything they need to consider, that allows them to check off what has been done, what is left to be done, and providing a definition that can be used to define and report upon governance along the way.
- Cycles — Provide a variety of cycles that every service will need to go to before they will be production worthy, forcing developers to iterate upon their services and harden and mature them before they will be considered ready for production.
- Reviews — Require all services go through a series of lifecycle reviews by other teams, pushing service owners to present their work to each review team and work with them to satisfy any concerns and make sure it meets all governance criteria.
- Clinics — Providing developers with a variety of clinics where they can receive feedback on their work, improve upon their service, and improve the health of their work before submitting it for inclusion in a production environment.
Enterprise organizations that provide structure for API development teams find it much easier to realize their governance aspirations. The scaffolding is already there to think about the consistency of services, and the face-to-face and virtual scrutiny of services help provide the environment for governance practices to be executed, enforced, and evolved before any service reaches a production state. A well-defined API deployment lifecycle will help contribute to a well-defined API governance practice.
One sign of enterprise groups who are further along in their governance journeys is when there are virtualized environments being put to use. Requiring all API developers to mock and iterate upon their APIs in a virtualized environment, presenting them as if they are real before they ever are given a license to write any code, let alone reach a production state for their services.
- Mocking — Creating mock APIs for all endpoints, virtualizing every aspect of the surface area of an API, allowing a service to be iterated upon early on in its lifecycle.
- Data — Requiring virtualized and synthesized data be present for all mocked APIs, returning realistic data with responses, reflecting behavior encountered in a production environment.
- Sandbox — Providing a complete lab and sandbox environment for developers to publish their mocked APIs into, reflecting what they’ll encounter in a production environment, but done in a much more safer and secure way.
Virtualized environments provide an important phase in the journey for APIs moving from concept to reality. Establishing a safe environment for developers to iterate upon their work, encounter many of the challenges they’ll face in a public environment, without any of the potentials for harm to users or the platform. Ensuring that when a service is ready for development, most of the rough edges have been worked out of the service contract, and for the team behind.
One of the most significant ways in which we’ve found enterprise groups governing the evolution of their APIs is through the technology they employ. This technology is providing much of the structure and discipline that organizations are depending on to help ensure that APIs are being developed and ultimately deployed in a consistent manner. Bringing most of the governance to the table for some organizations who haven’t begun moving their governance strategy forward as a formal approach.
- Authentication — Requiring standard-based approaches to authentication using Basic Auth, API keys, OAuth, and JWT. Ensuring the teams understand when to use which protocol and how to properly configure and use as part of larger API governance strategy.
- Framework — Relying on the programming frameworks in use to inject discipline into the process, dictating the governance of how APIs are delivered before they are ready for a production environment.
- Gateway — Applying the policies and structure necessary to govern API services as they are made available in a production environment. Many groups also had a sandbox or development edition of their gateway emulating many of the same elements that will be found in a production world.
- Management — Similar to the gateway, groups are relying on their API management layers to help govern what APIs do, providing transformations, policy templates, and a wide variety of other standardization that occurs prior to being made available in a production sense.
- Vendor — The reliance on technology to deliver governance at the API deployment level gives a lot of control to vendors when it comes to governing the API lifecycle. If a vendor doesn’t provide a specific way of doing things, it may not exist within some groups. Dictating what is governance for many enterprise groups.
- Tooling — Most groups have an arsenal of open source and custom-developed tooling for helping push code from development to production, validating, scanning, transforming, shaping, and hardening code and interfaces to be ready for production usage.
- Encryption — Requiring encryption by default for storage, and in transport, using technology to ensure security is a default parameter for everything that is exposed publicly. Reducing the possibility of a breach, and minimizing the damage when one does occur.
Demonstrating how important the technological choices we make, and the architectural planning we discussed earlier is to the overall API governance conversations. The services, tooling, and applications we adopt will either contribute or will not contribute to our governance practices. Potentially enforcing governance for all APIs as they move from development to production, in a way that teams cannot circumvent, and often times don’t even notice is occurring behind the scenes.
Augmenting the core technology, there are a number of orchestration practices we found that help quantify and enforce governance on the road from development to production. Dictating how code, artifacts, and other elements included as part of the API lifecycle move forward, evolve, or possibly get held back until specific requirements are met to meet wider governance criteria.
- Pipeline — CI/CD services, tooling, and mindset have introduced a pipeline workflow for many teams, standardizing the API delivery process as an executable, repeatable, measurable pipeline that can be realized across any team.
- Stages — The defining of clear stages that exist after development, but before production, requiring quality assurance, security reviews, compliance audits, and other relevant governance practices to be realized.
- Hooks — Well defined pre and post-commit hooks for all service repositories, requiring that governance is applied throughout a service’s pipeline, and are the default for all services, no matter which organization they emerge from.
- Devops — Pushing that all teams are competent and skilled enough to execute on behalf of their services from beginning to end, owning and executing at every stage of the life cycle. Reducing the need for reliance on special teams, and eliminating bottlenecks.
- Logging — Identifying the logging capabilities of the entire stack for each service being delivered. Making sure logging is turned on for everything, and all logs are shipped to a central location for auditing, and when possible real-time analysis and response.
Orchestration provides some clarity on the automation side of moving services from development to production, while also enforcing governance along the way. Allowing for an assembly line delivery of consistent services, and the iteration of each version, in alignment with the overall governance strategy. Reducing the chance for human error and increasing the chance for consistent execution of the enterprise API strategy at scale across many different teams.
The Legal Department
Beyond the technology, the legal department should have a significant influence over APIs going from development to production. Providing a structured framework that can generally apply across all services easily, but also providing granular level control over the fine-tuning of legal documents for specific services and use cases. With a handful of clear building blocks in use to help govern the delivery of APIs from the legal side of the equation.
- Terms of Services — Have universally applicable and modular as well as possibly machine-readable and human-readable terms of service governing all services from a central location.
- Security Policy — Provide a comprehensive security policy that governs how services are secured, reflecting the technologies, checklists, tooling, and reviews that are in use by all team members, providing an overview for all stakeholders to understand.
- Licensing — Establish clear code, data, and interface licensing to be used across the entire API stack, allowing developers to properly license their services, as well as understand the provenance for the systems and services they depend on.
- Server Level Agreements — Have universally applicable, modular, as well as possibly machine readable, and human-readable service level agreement (SLA) that can be applied across all services, measured, and reported upon as part of wider governance strategy.
- Scopes — Define and publish the OAuth access scopes as part of a formal set of legal guidance regarding what data is accessible via services, and the role users and developers play in managing scope that is defined by the platform.
The legal department will play an important role in governing APIs as they move from development to production, and there needs to be clear guidance for all developers regarding what is required. Similar to the technical and business elements of delivering services, the legal components need to be easy to access, understand, and apply, but also make sure and protect the interests of everyone involved with the delivery and operation of enterprise services.
Stay tuned for part six: Making APIs Available to Consumers.
Opinions expressed by DZone contributors are their own.