Generating AVRO Schemas for Data and Making Sure Names Are Correct
Generating AVRO Schemas for Data and Making Sure Names Are Correct
Learn how to use Apache NiFi to generate AVRO schemas while ensuring that the field names meet strict naming conventions.
Join the DZone community and get the full member experience.
Join For FreeThe open source HPCC Systems platform is a proven, easy to use solution for managing data at scale. Visit our Easy Guide to learn more about this completely free platform, test drive some code in the online Playground, and get started today.
Building schemas is tedious work and is often fraught with errors. The InferAvroSchema processor can get you started. It generates a compliant schema for use. There is one caveat: you have to make sure you are using Apache Avro-safe field names. I have a custom processor that will clean your attributes if you need them to be Avro-safe. See the processor listed below.
Example flow utilizing InferAvroSchema:
InferAvroSchema details:
The steps are as follows:
Use Apache NiFi to convert data to JSON or CSV.
Send JSON or CSV data to
InferAvroSchema. I recommend setting the output destination toflowfile-attribute, input content type tojson, and the pretty Avro output totrue.The new schema is now in the following attribute format:
inferred.avro.schema.
inferred.avro.schema
{ "type" : "record", "name" : "schema1",
"fields" : [ {
"name" : "table", "type" : "string",
"doc" : "Type inferred from '\"schema1.tableName\"'" } ]
} This schema can then be used for conversions directly or can be stored in Hortonworks Schema Registry or Apache NiFi's built-in Avro Registry.
Now, you can use it for ConvertRecord, QueryRecord, and other Record processing.
Example generated schema in Avro-JSON format stored in Hortonworks Schema Registry:
And that's it!
Managing data at scale doesn’t have to be hard. Find out how the completely free, open source HPCC Systems platform makes it easier to update, easier to program, easier to integrate data, and easier to manage clusters. Download and get started today.
Like This Article? Read More From DZone
Opinions expressed by DZone contributors are their own.
{{ parent.title || parent.header.title}}
{{ parent.tldr }}
{{ parent.linkDescription }}
{{ parent.urlSource.name }}