✳ OMNIVIEWER Parquet viewer CSV viewer ← back

How to read a 20 GB Parquet file

A 20 GB Parquet file will never fit in memory — and yet you can read its full schema and every column's statistics in milliseconds, without a single row scan. The trick is the format itself: Parquet is columnar and footer-indexed. This is a tour of how it's laid out on disk, then the footer-first algorithm that opens a file far larger than RAM, with real JavaScript you can lift.

On this page What is Parquet The file layout The footer Why 20 GB is easy Encoding & compression Decoding rows Caveats

What is Parquet?

Apache Parquet is a binary, columnar storage format for analytical data. It was created in 2013 by engineers at Twitter and Cloudera — inspired by Google's Dremel paper — and donated to the Apache Software Foundation. Today it's the storage backbone of the data-lake world: Spark, pandas, Polars, DuckDB, Apache Arrow, BigQuery, Snowflake and Athena all read and write it.

The key idea is in the word columnar. A CSV or a JSON-lines file is row-major: all of row 1's fields, then all of row 2's. Parquet is column-major: all of column A's values, then all of column B's. That single change buys two things. First, values in a column are all the same type and tend to be similar, so they compress dramatically better. Second, a query that needs three of your fifty columns can read just those three — column pruning — instead of dragging the whole row off disk.

The file layout

A Parquet file is a sandwich. It opens and closes with the 4-byte magic PAR1, and everything between is column data followed by a metadata footer:

┌──────────┐  4 bytes: "PAR1"
│  PAR1    │
├──────────┤  ─┐
│ RowGroup │   │  Row group 0
│  col A   │   │    · column chunk A  (its pages)
│  col B   │   │    · column chunk B
│   ...    │   │
├──────────┤   ├─ ... more row groups ...
│ RowGroup │   │
│   ...    │   │
├──────────┤  ─┘
│ FileMeta │   Thrift-encoded FileMetaData (the footer)
├──────────┤  4 bytes: footer length (little-endian uint32)
│  len     │
├──────────┤  4 bytes: "PAR1"
│  PAR1    │
└──────────┘

The nesting is the whole model, so it's worth naming each layer:

Crucially, none of the column data is self-describing on its own — you can't point at byte 5,000,000 and know what's there. The map lives entirely in the footer.

The last 8 bytes of every Parquet file are a 4-byte little-endian footer length followed by PAR1. Just before them sits the FileMetaData structure, which records everything: the schema, the number of rows, and for every row group, every column chunk's byte offset, size, compression codec, encodings and min/max statistics.

FileMetaData is encoded with the Thrift compact protocol — a compact, self-describing binary encoding where each struct field carries a small header (a 4-bit type and a 4-bit id delta from the previous field), integers are zig-zag varints, and there's no external schema needed to walk it. Here's the shape of what it holds, in Thrift IDL:

struct FileMetaData {
  1: i32               version
  2: list<SchemaElement> schema      // the columns, as a flat pre-order tree
  3: i64               num_rows
  4: list<RowGroup>    row_groups
  5: list<KeyValue>    key_value_metadata   // e.g. the pandas / Arrow block
  6: string            created_by
}
struct ColumnMetaData {
  1: Type    type            // BOOLEAN, INT32, INT64, DOUBLE, BYTE_ARRAY, ...
  4: CompressionCodec codec  // UNCOMPRESSED, SNAPPY, GZIP, ZSTD, ...
  5: i64     num_values
  6: i64     total_uncompressed_size
  7: i64     total_compressed_size
  9: i64     data_page_offset
  11:i64     dictionary_page_offset
  12:Statistics statistics   // min, max, null_count
}

Because it's self-describing, a reader for the compact protocol is tiny — a few dozen lines that return each struct as a map of field-id → value:

function readStruct(buf, r) {         // r = { pos }
  const out = {};
  let lastId = 0;
  for (;;) {
    const header = buf[r.pos++];
    if (header === 0) break;           // STOP
    const type = header & 0x0f;
    const delta = header >> 4;
    const id = delta === 0 ? zigzag(varint(buf, r)) : lastId + delta;
    lastId = id;
    out[id] = readValue(buf, r, type); // int / double / binary / list / struct
  }
  return out;
}

Why 20 GB is easy

Here's the payoff. To learn a Parquet file's schema and per-column statistics, you never read the data. You read the last few bytes to find the footer length, then read the footer:

async function readParquetFooter(file) {          // file is a Blob/File handle
  const tail = new Uint8Array(await file.slice(file.size - 8, file.size).arrayBuffer());
  const dv = new DataView(tail.buffer);
  const magic = String.fromCharCode(...tail.slice(4));
  if (magic !== 'PAR1') throw new Error('not a Parquet file');
  const footerLen = dv.getUint32(0, true);        // little-endian

  const start = file.size - 8 - footerLen;         // seek straight to it
  const footerBytes = new Uint8Array(await file.slice(start, file.size - 8).arrayBuffer());
  return readStruct(footerBytes, { pos: 0 });      // the whole FileMetaData
}

A File in the browser is a lazy handle — file.slice(a, b) doesn't read anything; only .arrayBuffer() touches the disk, and then only that range. So this is two small reads whether the file is 2 KB or 20 GB. That's the design principle behind the whole toolkit: OmniViewer's Parquet viewer paints the SCHEMA and ROW GROUPS tabs from this footer alone, and only reads column data when you open the TABLE preview.

Column pruning, for free. Because every column chunk records its own byte offset and size in the footer, a reader that wants one column out of fifty seeks straight to that column's bytes in each row group and skips the rest. That's why analytical queries on Parquet are so much cheaper than on CSV — and it's the same seek-don't-scan idea that makes the footer read instant.

Encoding & compression

Inside a data page, values aren't just dumped as raw bytes. Parquet layers two ideas that exploit the column-major layout.

Encoding shrinks the values structurally before any byte compression. The default for most columns is dictionary encoding: the distinct values are written once into a dictionary page, and the data page stores small integer indices into it. Those indices are then packed with the RLE / bit-packing hybrid — runs of a repeated index collapse to a count, and otherwise the indices are bit-packed at just the width needed (a column with 200 distinct values needs 8 bits per index, not 64). The same hybrid encodes definition levels, the per-value flags that mark which entries are null. Here's the core of decoding it:

function decodeHybrid(buf, pos, bitWidth, count) {
  const out = [];
  while (out.length < count) {
    const header = varint(buf, pos);          // advances pos
    if (header & 1) {                          // bit-packed run
      const groups = header >> 1;              // groups of 8 values
      for (const v of unpackBits(buf, pos, bitWidth, groups * 8)) out.push(v);
      pos.v += groups * bitWidth;              // bytes consumed
    } else {                                    // RLE run: a value repeated
      const runLen = header >> 1;
      const val = readLE(buf, pos, Math.ceil(bitWidth / 8));
      for (let i = 0; i < runLen; i++) out.push(val);
    }
  }
  return out;
}

Compression then runs a general codec over the encoded page. The default is Snappy — fast, moderate ratio — with GZIP and increasingly Zstd as alternatives. Snappy is a simple block format (a varint uncompressed length, then literal and back-reference "copy" tokens), so it decompresses by hand in a page of code — which matters in a browser, since no built-in primitive speaks Snappy the way DecompressionStream speaks gzip.

Decoding rows for a preview

Putting it together, a decoded preview of the first row group is: seek to each column chunk (its dictionary page offset, else its first data page offset, both in the footer), read a bounded prefix of it, then walk its pages — decompress each page body, decode the dictionary if present, decode the definition levels to know where the nulls are, decode the value indices or PLAIN values, and interleave them:

// per data page, for a flat (non-nested) column:
const defLevels = maxDef ? decodeHybrid(page, pos, bitWidth(maxDef), numValues) : null;
const present   = defLevels ? defLevels.filter(d => d === maxDef).length : numValues;
const decoded   = encoding === PLAIN
  ? decodePlain(page, pos, type, present)
  : indices(page, pos, present).map(i => dictionary[i]);   // dictionary-encoded

let di = 0;                                                 // stitch nulls back in
for (let i = 0; i < numValues; i++)
  rows.push(defLevels && defLevels[i] !== maxDef ? null : decoded[di++]);

Because you only need a screenful of rows, you read only the first page or two of each column chunk — so the TABLE preview stays bounded even when a column chunk is gigabytes.

Caveats

A few honest edges. Nested and repeated columns (lists, maps, structs) use repetition levels as well as definition levels, and reassembling them is the trickier part of the spec — a flat, tabular file is the common case and the easy one. Some newer files use delta or byte-stream-split encodings, and codecs like Zstd, LZ4 and Brotli have no browser primitive — a viewer can flag those on the data preview while still reading the schema and statistics from the footer, which never depend on the codec. And a corrupt footer leaves you with only the raw bytes; unlike a row-major format, there's no reliable way to recover columnar data without the index.

None of that changes the headline, though: Parquet's footer-first, columnar design is exactly what lets a static web page open a file far bigger than memory and tell you what's inside — instantly.

Open a Parquet file →Read the schema, row groups, column statistics and a data preview — 100% in your browser, no upload. All formats →Every file type OmniViewer can open, and what each toolkit does.