A Schema Architecture for Microservices
A new microservices challenger appears.
Join the DZone community and get the full member experience.Join For Free
Message and Event payload validation has been a rather thorny problem ever since extensible data structures (XML, JSON, YAML...) started to be used at scale. In fact, very little progress has been made since the good old days of DTDs. Schema definition languages such as XML-schema, json-schema, or even the OpenAPI schema are unfamiliar to most developers and often result in a rather anemic validation set of rules, leading to a perceived low value, and therefore a lack of interest.
There are three key problems in a schema architecture:
- the lack of reusability of data structure definitions
- the lack of expressivity of the schema language
- the developer experience
The catch with any schema language is that it can be used to express a data model or a message type definition, but it cannot be used to do both at the same time. This has led to the emergence of crazy patterns like the salami slice or the garden of Eden. The reality is that the problem of expressing a common data model from which message and event types can be defined requires two domain-specific languages: a schema language and a projection language that expresses which element of the common data model participate in the message type. Bonus points if that projection language can express sophisticated validation rules in relation to the common data model.
It happens that there is actually a schema language that comes with an associated projection language and that schema language is used profusely in modern microservice architectures, namely GraphQL and its associated query language. Pretty much every developer knows and loves GraphQL, and if not it's super easy to learn! GraphQL comes with a large number of tools to edit, validate and document schemas and queries. Cherry on the cake, GraphQL has one of the best and intuitive DevX around.
At Nav, we got the idea of using that foundation to develop our Schema Architecture (it is now a FOSS project). It's being used in production today and there is no turning back. We get consistent validation across all of the nodes of our microservice architecture, something that we never achieved before. The development gains are significant when you compare the message or event definition to the generated schemas and code libraries. In any case, that code is tedious to write and very hard to keep in sync across languages.
How does it work? First, you define your common data model using GraphQL schema language (as a side note, there is no GraphQL runtime, we are only using GraphQL as a syntax from which we generate artifacts). Here is an example based on Tweeter's data model:
Then you define a message as a projection of the common data model (note the order directive is only required when you use protobuf):
This work was made possible by Brent Tubbs, Senior Manager of Engineering, who gave us the resources to build the project, and a team of passionate engineers: Michal Scienski, Daniel Zemichael, Jeff Warner, Jovon McCloud, and myself. Please take a look also at the build pipeline that Michal implemented, that's a major achievement.
Published at DZone with permission of Jean-Jacques Dubray. See the original article here.
Opinions expressed by DZone contributors are their own.