Writing Parquet Format Data to Regular Files (i.e., Not Hadoop HDFS)

The Apache Parquet format is a compressed, efficient columnar data representation. The existing Parquet Java libraries available were developed for and within the Hadoop ecosystem. Hence there tends to a be near automatic assumption that one is working with the Hadoop distributed filesystem, HDFS.

There are situations that one might want to create Parquet-formatted data to a regular file system file - particularly if not working in a context that assumes Hadoop and HDFS are present. Some big data tools and runtime stacks, which do not assume Hadoop, can work directly with Parquet files.

Recently I was tasked with being able to generate Parquet formatted data files into a regular file system and so set out to find example code of how to go about writing Parquet files. Most examples I came up with did so in the context of Hadoop HDFS. I found this one ParquetReaderWriterWithAvro, that alluded to the possibility of creating Parquet as a regular file, but tended to be shy of some of the crucial specifics.

Nonetheless, I went on and worked with the examples I found and figured out the details of how to make it all work - round trip write data into a Parquet regular file and then read it back.

I noticed that others had an interest in this as well and so decided to clean up my test bed project a bit, make it open source under the MIT license, and put it on public GitHub:

avro2parquet - Example program that writes Parquet formatted data to plain files (i.e., not Hadoop HDFS); Parquet is a columnar storage format.

Here, in this Maven-built Java 8 project, you can see all the details that are necessary to make this work out of the box. For instance, I have figured out the necessary pom file dependencies that work with the latest release of the Parquet libraries: parquet-hadoop and parquet-avro. The README file will mention a few other gotchas, such as needing to define the environment variable HADOOP_HOME.

The crucial information, though, is how to implement one's own versions of org.apache.parquet.io.OutputFile and org.apache.parquet.io.PositionOutputStream for writing to a Parquet output stream and org.apache.parquet.io.InputFile and org.apache.parquet.io.SeekableInputStream for reading from a Parquet stream. The builder for org.apache.parquet.avro.AvroParquetWriter accepts an OutputFile instance whereas the builder for org.apache.parquet.avro.AvroParquetReader accepts an InputFile instance.

This example illustrates writing Avro format data to Parquet. Avro is a row or record-oriented serialization protocol (that is, not columnar-oriented). The nice thing about Avro is that its schema for objects can be composed dynamically at runtime if need be. It should be fairly straightforward to put a JSON object, or CSV row, into an Avro representation and then write it out via the AvroParquetWriter. As they say, that is an exercise left for the reader.