Parquet

Binary Protocol Extensions

Mon, 01 Jan 0001 00:00:00 +0000

Binary Protocol Extensions

The extension mechanism of the binary Thrift field-id 32767 has some desirable properties:

Existing readers will ignore these extensions without any modifications
Existing readers will ignore the extension bytes with little processing overhead
The content of the extension is freeform and can be encoded in any format. This format is not restricted to Thrift.
Extensions can be appended to existing Thrift serialized structs without requiring Thrift libraries for manipulation (or changes to the thrift IDL).

Because only one field-id is reserved the extension bytes themselves require disambiguation; otherwise readers will not be able to decode extensions safely. This is left to implementers which MUST put enough unique state in their extension bytes for disambiguation. This can be relatively easily achieved by adding a UUID at the start or end of the extension bytes. The extension does not specify a disambiguation mechanism to allow more flexibility to implementers.

Compression

Mon, 01 Jan 0001 00:00:00 +0000

Parquet compression definitions

This document contains the specification of all supported compression codecs.

Overview

Parquet allows the data block inside dictionary pages and data pages to be compressed for better space efficiency. The Parquet format supports several compression codecs covering different areas in the compression ratio / processing cost spectrum.

The detailed specifications of compression codecs are maintained externally by their respective authors or maintainers, which we reference hereafter.

For all compression codecs except the deprecated LZ4 codec, the raw data of a (data or dictionary) page is fed as-is to the underlying compression library, without any additional framing or padding. The information required for precise allocation of compressed and decompressed buffers is written in the PageHeader struct.

Encodings

Mon, 01 Jan 0001 00:00:00 +0000

Parquet encoding definitions

This file contains the specification of all supported encodings.

Unless otherwise stated in page or encoding documentation, any encoding can be used with any page type.

Supported Encodings

For details on current implementation status, see the Implementation Status page.

Encoding type	Encoding enum	Supported Types
Plain	PLAIN = 0	All Physical Types
Dictionary Encoding	PLAIN_DICTIONARY = 2 (Deprecated) RLE_DICTIONARY = 8	All Physical Types
Run Length Encoding / Bit-Packing Hybrid	RLE = 3	BOOLEAN, Dictionary Indices
Delta Encoding	DELTA_BINARY_PACKED = 5	INT32, INT64
Delta-length byte array	DELTA_LENGTH_BYTE_ARRAY = 6	BYTE_ARRAY
Delta Strings	DELTA_BYTE_ARRAY = 7	BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY
Byte Stream Split	BYTE_STREAM_SPLIT = 9	INT32, INT64, FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY

Deprecated Encodings

Encoding type	Encoding enum
Bit-packed (Deprecated)	BIT_PACKED = 4

Encryption

Mon, 01 Jan 0001 00:00:00 +0000

Parquet Modular Encryption

Parquet files containing sensitive information can be protected by the modular encryption mechanism that encrypts and authenticates the file data and metadata - while allowing for a regular Parquet functionality (columnar projection, predicate pushdown, encoding and compression).

1 Problem Statement

Existing data protection solutions (such as flat encryption of files, in-storage encryption, or use of an encrypting storage client) can be applied to Parquet files, but have various security or performance issues. An encryption mechanism, integrated in the Parquet format, allows for an optimal combination of data security, processing speed and encryption granularity.

License

Mon, 01 Jan 0001 00:00:00 +0000

License

Spark Summit 2020

Mon, 01 Jan 0001 00:00:00 +0000

Slides

Sub-Projects

Mon, 01 Jan 0001 00:00:00 +0000

The parquet-format project contains format specifications and Thrift definitions of metadata required to properly read Parquet files.

The parquet-java project is a Java library to read and write Parquet files. It consists of multiple sub-modules, which implement the core components of reading and writing a nested, column-oriented data stream, to and from the Parquet format, along with Hadoop Input/Output Formats, Pig loaders, and other Java-based utilities for interacting with Parquet.

The parquet-cpp project is a C++ library to read-write Parquet files. It is part of the Apache Arrow C++ implementation, with bindings to Python, R, Ruby and C/GLib.

Building Parquet

Mon, 01 Jan 0001 00:00:00 +0000

Building Java resources can be build using mvn package. The current stable version should always be available from Maven Central.

C++ thrift resources can be generated via make.

Thrift can be also code-genned into any other thrift-supported language.

Hadoop Summit 2014

Mon, 01 Jan 0001 00:00:00 +0000

Slides

Motivation

Mon, 01 Jan 0001 00:00:00 +0000

We created Parquet to make the advantages of compressed, efficient columnar data representation available to any project in the Hadoop ecosystem.

Parquet is built from the ground up with complex nested data structures in mind, and uses the record shredding and assembly algorithm described in the Dremel paper. We believe this approach is superior to simple flattening of nested name spaces.

Parquet is built to support very efficient compression and encoding schemes. Multiple projects have demonstrated the performance impact of applying the right compression and encoding scheme to the data. Parquet allows compression schemes to be specified on a per-column level, and is future-proofed to allow adding more encodings as they are invented and implemented.

Security

Mon, 01 Jan 0001 00:00:00 +0000

Security

#CONF 2014

Mon, 01 Jan 0001 00:00:00 +0000

Contributing to Parquet-Java

Mon, 01 Jan 0001 00:00:00 +0000

Pull Requests

We prefer to receive contributions in the form of GitHub pull requests. Please send pull requests against the github.com/apache/parquet-java repository. If you’ve previously forked Parquet from its old location, you will need to add a remote or update your origin remote to https://github.com/apache/parquet-java.git. Here are a few tips to get your contribution in:

Break your work into small, single-purpose patches if possible. It’s much harder to merge in a large change with a lot of disjoint features.
Create an Issue on the Parquet-Java issues.
Submit the patch as a GitHub pull request against the master branch. For a tutorial, see the GitHub guides on forking a repo and sending a pull request. Prefix your pull request name with the Issue GH-2935: (ex: https://github.com/apache/parquet-java/pull/2951).
Make sure that your code passes the unit tests. You can run the tests with mvn test in the root directory.
Add new unit tests for your code.
All Pull Requests are tested automatically on GitHub Actions.

If you’d like to report a bug but don’t have time to fix it, you can still raise an issue, or email the mailing list (dev@parquet.apache.org).

Sponsor

Mon, 01 Jan 0001 00:00:00 +0000

Sponsor

Donate

Mon, 01 Jan 0001 00:00:00 +0000

Donate

Releasing Parquet-Java

Mon, 01 Jan 0001 00:00:00 +0000

Setup

N.B. The mechanics of releasing parquet-format is the same (e.g. setting up keys, branching, votes, etc)

You will need:

PGP code signing keys, published in KEYS.
Permission to stage artifacts in Nexus.

Make sure you have permission to deploy Parquet artifacts to Nexus by pushing a snapshot:

mvn deploy

If you have problems, read the publishing Maven artifacts documentation.

Release process

Parquet uses the maven-release-plugin to tag a release and push binary artifacts to staging in Nexus. Once maven completes the release, the official source tarball is built from the tag.

Strata 2013

Mon, 01 Jan 0001 00:00:00 +0000

Slides

Configurations

Mon, 01 Jan 0001 00:00:00 +0000

Row Group Size

Larger row groups allow for larger column chunks which makes it possible to do larger sequential IO. Larger groups also require more buffering in the write path (or a two pass write). We recommend large row groups (512MB - 1GB). Since an entire row group might need to be read, we want it to completely fit on one HDFS block. Therefore, HDFS block sizes should also be set to be larger. An optimized read setup would be: 1GB row groups, 1GB HDFS block size, 1 HDFS block per HDFS file.

Events

Mon, 01 Jan 0001 00:00:00 +0000

Events

Extensibility

Mon, 01 Jan 0001 00:00:00 +0000

There are many places in the format for compatible extensions:

File Version: The file metadata contains a version.
Encodings: Encodings are specified by enum and more can be added in the future.
Page types: Additional page types can be added and safely skipped.

Geospatial Type

Mon, 01 Jan 0001 00:00:00 +0000

Geospatial Definitions

This document contains the specification of geospatial types and statistics.

Background

The Geometry and Geography class hierarchy and its Well-Known Text (WKT) and Well-Known Binary (WKB) serializations (ISO variant supporting XY, XYZ, XYM, XYZM) are defined by OpenGIS Implementation Specification for Geographic information - Simple feature access - Part 1: Common architecture, from OGC(Open Geospatial Consortium).

The version of the OGC standard first used here is 1.2.1, but future versions may also be used if the WKB representation remains wire-compatible.

Logical Types

Mon, 01 Jan 0001 00:00:00 +0000

Parquet Logical Type Definitions

Logical types are used to extend the types that parquet can be used to store, by specifying how the primitive types should be interpreted. This keeps the set of primitive types to a minimum and reuses parquet’s efficient encodings. For example, strings are stored with the primitive type BYTE_ARRAY with a STRING annotation.

This file contains the specification for all logical types.

Metadata

The parquet format’s LogicalType stores the type annotation. The annotation may require additional metadata fields, as well as rules for those fields.

Metadata

Mon, 01 Jan 0001 00:00:00 +0000

There are two types of metadata: file metadata, and page header metadata.

All thrift structures are serialized using the TCompactProtocol. The full definition of these structures is given in the Parquet Thrift definition.

File metadata

In the diagram below, file metadata is described by the FileMetaData structure. This file metadata provides offset and size information useful when navigating the Parquet file.

Page header metadata (PageHeader and children in the diagram) is stored in-line with the page data, and is used in the reading and decoding of data.

Variant Shredding

Mon, 01 Jan 0001 00:00:00 +0000

Variant Shredding

The Variant type is designed to store and process semi-structured data efficiently, even with heterogeneous values. Query engines encode each Variant value in a self-describing format, and store it as a group containing value and metadata binary fields in Parquet. Since data is often partially homogeneous, it can be beneficial to extract certain fields into separate Parquet columns to further improve performance. This process is called shredding.

Shredding enables the use of Parquet’s columnar representation for more compact data encoding, column statistics for data skipping, and partial projections.

Variant Type

Mon, 01 Jan 0001 00:00:00 +0000

Variant Binary Encoding

A Variant represents a type that contains one of:

Primitive: A type and corresponding value (e.g. INT, STRING)
Array: An ordered list of Variant values
Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys.

A Variant is encoded with 2 binary values, the value and the metadata.

There are a fixed number of allowed primitive types, provided in the table below. These represent a commonly supported subset of the logical types allowed by the Parquet format.

Nested Encoding

Mon, 01 Jan 0001 00:00:00 +0000

To encode nested columns, Parquet uses the Dremel encoding with definition and repetition levels. Definition levels specify how many optional fields in the path for the column are defined. Repetition levels specify at what repeated field in the path has the value repeated. The max definition and repetition levels can be computed from the schema (i.e. how much nesting there is). This defines the maximum number of bits required to store the levels (levels are defined for all values in the column).

Bloom Filter

Mon, 01 Jan 0001 00:00:00 +0000

Parquet Bloom Filter

Problem statement

In their current format, column statistics and dictionaries can be used for predicate pushdown. Statistics include minimum and maximum value, which can be used to filter out values not in the range. Dictionaries are more specific, and readers can filter out values that are between min and max but not in the dictionary. However, when there are too many distinct values, writers sometimes choose not to add dictionaries because of the extra space they occupy. This leaves columns with large cardinalities and widely separated min and max without support for predicate pushdown.

Checksumming

Mon, 01 Jan 0001 00:00:00 +0000

Pages of all kinds can be individually checksummed. This allows disabling of checksums at the HDFS file level, to better support single row lookups. Checksums are calculated using the standard CRC32 algorithm - as used in e.g. GZip - on the serialized binary representation of a page (not including the page header itself).

Column Chunks

Mon, 01 Jan 0001 00:00:00 +0000

Column chunks are composed of pages written back to back. The pages share a common header and readers can skip over pages they are not interested in. The data for the page follows the header and can be compressed and/or encoded. The compression and encoding is specified in the page metadata.

A column chunk might be partly or completely dictionary encoded. It means that dictionary indexes are saved in the data pages instead of the actual values. The actual values are stored in the dictionary page. See details in Encodings.md. The dictionary page must be placed at the first position of the column chunk. At most one dictionary page can be placed in a column chunk.

Error Recovery

Mon, 01 Jan 0001 00:00:00 +0000

If the file metadata is corrupt, the file is lost. If the column metadata is corrupt, that column chunk is lost (but column chunks for this column in other row groups are okay). If a page header is corrupt, the remaining pages in that chunk are lost. If the data within a page is corrupt, that page is lost. The file will be more resilient to corruption with smaller row groups.

Nulls

Mon, 01 Jan 0001 00:00:00 +0000

Nullity is encoded in the definition levels (which is run-length encoded). NULL values are not encoded in the data. For example, in a non-nested schema, a column with 1000 NULLs would be encoded with run-length encoding (0, 1000 times) for the definition levels and nothing else.

Page Index

Mon, 01 Jan 0001 00:00:00 +0000

Parquet page index: Layout to Support Page Skipping

In Parquet, a page index is optional metadata for a ColumnChunk, containing statistics for DataPages that can be used to skip those pages when scanning in ordered and unordered columns. The page index is stored using the OffsetIndex and ColumnIndex structures, defined in parquet.thrift

Problem Statement

In previous versions of the format, Statistics are stored for ColumnChunks in ColumnMetaData and for individual pages inside DataPageHeader structs. When reading pages, a reader had to process the page header to determine whether the page could be skipped based on the statistics. This means the reader had to access all pages in a column, thus likely reading most of the column data from disk.

Implementation status

Mon, 01 Jan 0001 00:00:00 +0000

This page summarizes the features supported by different Parquet implementations.

Note: If you find out of date information, please help us improve the accuracy of this page by opening an issue or submitting a pull request.

Legend

The value in each box means:

✅: supported. Footnote added when support is partial. When data is available, links to release notes are provided on the implementing version.
❌: not supported
(R): only read support
(W): only write support
(blank): no data

Implementations

arrow (C++)
parquet-java (Java)
arrow-go (Go)
arrow-rs (Rust)
cudf (cuDF C++)
hyparquet (JavaScript)
duckdb (C++)
polars (Rust)

Physical types

Physical types are defined by the enum Type in parquet.thrift

Variant Type in Apache Parquet for Semi-Structured Data

Fri, 27 Feb 2026 00:00:00 +0000

The Apache Parquet community is excited to announce the addition of the Variant type—a feature that brings native support for semi-structured data to Parquet, significantly improving efficiency compared to less efficient formats such as JSON. This marks a significant addition to Parquet, demonstrating how the format continues to evolve to meet modern data engineering needs.

While Apache Parquet has long been the standard for structured data where each value has a fixed and known type, handling heterogeneous, nested data often required a compromise: either store it as a costly-to-parse JSON string or flatten it into a rigid schema. The introduction of the Variant logical type provides a native, high-performance solution for semi-structured data that is already seeing rapid uptake across the ecosystem.

Native Geospatial Types in Apache Parquet

Fri, 13 Feb 2026 00:00:00 +0000

Geospatial data has become a core input for modern analytics across logistics, climate science, urban planning, mobility, and location intelligence. Yet for a long time, spatial data lived outside the mainstream analytics ecosystem. In primarily non-spatial data engineering workflows, spatial data was common but required workarounds to handle efficiently at scale. Formats such as Shapefile, GeoJSON, or proprietary spatial databases worked well for visualization and GIS workflows, but they did not integrate cleanly with large scale analytical engines.

1.17.0

Tue, 13 Jan 2026 00:00:00 +0000

The latest version of parquet-java is 1.17.0.

For the changes, please check out the Release on Github.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.16.0.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

1.16.0

Wed, 03 Sep 2025 00:00:00 +0000

The latest version of parquet-java is 1.16.0.

For the changes, please check out the Release on Github.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.15.2.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

2.12.0

Thu, 28 Aug 2025 00:00:00 +0000

Github Release Link.

The latest version of parquet-format is 2.12.0.

To check the validity of this release, use its:

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.15.2

Thu, 01 May 2025 00:00:00 +0000

The latest version of parquet-java is 1.15.2.

For the changes, please check out the Release on Github.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.14.4.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

2.11.0

Sun, 23 Mar 2025 00:00:00 +0000

Github Release Link.

The latest version of parquet-format is 2.11.0.

To check the validity of this release, use its:

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.15.1

Sun, 16 Mar 2025 00:00:00 +0000

The latest version of parquet-java is 1.15.1.

For the changes, please check out the Release on Github.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.14.4.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

1.15.0

Mon, 02 Dec 2024 00:00:00 +0000

The latest version of parquet-java is 1.15.0.

For the changes, please check out the Release on Github.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.15.0.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

1.14.4

Mon, 11 Nov 2024 00:00:00 +0000

The latest version of parquet-java is 1.14.4.

With the following bugfixes:

GH-3040: DictionaryFilter.canDrop may return false positive result when dict size exceeds 8k
GH-3029: Fix EncryptionPropertiesHelper not to use java.nio.file.Path
GH-3042: Throw exception in BytesInput

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.13.1.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

1.14.3

Sat, 05 Oct 2024 00:00:00 +0000

The latest version of parquet-java is 1.14.3.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.13.1.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

Add the following dependency section to your pom.xml:

 <dependencies>
 ...
 <dependency>
 <groupId>org.apache.parquet</groupId>
 <artifactId>parquet-avro</artifactId>
 <version>1.14.3</version> <!-- or latest version -->
 </dependency>
 ...
 </dependencies>

Older Releases

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.14.2

Mon, 26 Aug 2024 00:00:00 +0000

The latest version of parquet-java is 1.14.2.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.13.1.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

Add the following dependency section to your pom.xml:

 <dependencies>
 ...
 <dependency>
 <groupId>org.apache.parquet</groupId>
 <artifactId>parquet-avro</artifactId>
 <version>1.14.2</version> <!-- or latest version -->
 </dependency>
 ...
 </dependencies>

Older Releases

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.14.1

Sun, 16 Jun 2024 00:00:00 +0000

The latest version of parquet-java is 1.14.1.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.13.1.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

Add the following dependency section to your pom.xml:

 <dependencies>
 ...
 <dependency>
 <groupId>org.apache.parquet</groupId>
 <artifactId>parquet-avro</artifactId>
 <version>1.14.1</version> <!-- or latest version -->
 </dependency>
 ...
 </dependencies>

Older Releases

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.14.0

Tue, 07 May 2024 00:00:00 +0000

The latest version of parquet-java is 1.14.0.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.13.1.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

Add the following dependency section to your pom.xml:

 <dependencies>
 ...
 <dependency>
 <groupId>org.apache.parquet</groupId>
 <artifactId>parquet-avro</artifactId>
 <version>1.14.0</version> <!-- or latest version -->
 </dependency>
 ...
 </dependencies>

Older Releases

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

2.10.0

Mon, 20 Nov 2023 00:00:00 +0000

Github Release Link.

The latest version of parquet-format is 2.10.0.

To check the validity of this release, use its:

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.13.1

Thu, 18 May 2023 00:00:00 +0000

The latest version of parquet-java is 1.13.1.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.12.3.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

Add the following dependency section to your pom.xml:

 <dependencies>
 ...
 <dependency>
 <groupId>org.apache.parquet</groupId>
 <artifactId>parquet-avro</artifactId>
 <version>1.13.1</version> <!-- or latest version -->
 </dependency>
 ...
 </dependencies>

Older Releases

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.13.0

Thu, 06 Apr 2023 00:00:00 +0000

The latest version of parquet-java is 1.13.0.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.12.3.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

Add the following dependency section to your pom.xml:

 <dependencies>
 ...
 <dependency>
 <groupId>org.apache.parquet</groupId>
 <artifactId>parquet-avro</artifactId>
 <version>1.13.0</version> <!-- or latest version -->
 </dependency>
 ...
 </dependencies>

Older Releases

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.12.3

Thu, 26 May 2022 00:00:00 +0000

The latest version of parquet-java is 1.12.3.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.11.2.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

Add the following dependency section to your pom.xml:

 <dependencies>
 ...
 <dependency>
 <groupId>org.apache.parquet</groupId>
 <artifactId>parquet-avro</artifactId>
 <version>1.12.3</version> <!-- or latest version -->
 </dependency>
 ...
 </dependencies>

Older Releases

Older releases can be found in the Archives of the Apache Software Foundation: https://archive.apache.org/dist/parquet/

1.12.2

Wed, 06 Oct 2021 00:00:00 +0000

The latest version of parquet-java is 1.12.2.

To check the validity of this release, use its:

The latest version of parquet-java on the previous minor branch is 1.11.2.

To check the validity of this release, use its:

Downloading from the Maven central repository

The Parquet team publishes its releases to Maven Central.

Add the following dependency section to your pom.xml:

 <dependencies>
 ...
 <dependency>
 <groupId>org.apache.parquet</groupId>
 <artifactId>parquet-avro</artifactId>
 <version>1.12.2</version> <!-- or latest version -->
 </dependency>
 ...
 </dependencies>