Inside a gzip file

A .gz has no index. Not a small one, not a lazy one — none at all. Nothing in the file records where anything is, which is why gzip cannot seek, cannot decompress in parallel, and why almost nothing will open a big one in a browser. This is the format field by field, the DEFLATE bitstream inside it, and the walker we wrote that finds where a member ends without ever reconstructing its output — with real JavaScript you can lift.

What a .gz actually is

gzip was written by Jean-loup Gailly and Mark Adler in 1992, to give the GNU project a compressor unencumbered by the patents then hanging over compress. The file format is RFC 1952; the compression inside it is DEFLATE, RFC 1951. Both are still, three decades later, the most widely deployed compression on earth: every Content-Encoding: gzip response, every PNG scanline, every ZIP member and every .tar.gz release tarball is DEFLATE.

The first thing to understand is that a gzip file is not a container. It is a stream, defined as a sequence of independent members:

┌──────────────── member 1 ────────────────┐┌──── member 2 ────┐
│ header │ DEFLATE data │ CRC32 │ ISIZE    ││ header │ ... │ ...│
└──────────────────────────────────────────┘└──────────────────┘
  10+ B    variable        4 B     4 B

Each member is a complete gzip file in its own right. That has a consequence people rediscover constantly, usually by accident:

$ cat part1.gz part2.gz > both.gz
$ gunzip -c both.gz | wc -c        # works — you get both payloads

Concatenating two .gz files produces a valid third one. This is not a quirk being tolerated; it is the design, and tools lean on it. bgzip (from the samtools suite) and pigz -b compress in independent blocks precisely so that a reader can jump to any member without touching the ones before it — which is the only reason a 200 GB BAM alignment file can be indexed at all.

The member header, field by field

Ten fixed bytes, then whatever the flags say is present.

The FLG byte then decides what comes next, in this order:

Two details bite in practice. First, FNAME leaks: gzip data.csv writes data.csv into the header, so a file you renamed still carries its old name, and one compressed from a temp path can carry that. Second, the RFC says these strings are ISO‑8859‑1, but every archiver on a UTF‑8 system writes UTF‑8 anyway — so a reader has to try UTF‑8 strictly and fall back:

function decodeName(raw) {
  try { return new TextDecoder('utf-8', { fatal: true }).decode(raw); }
  catch { return new TextDecoder('latin1').decode(raw); }  // as the RFC says
}

FEXTRA is where the interesting extension lives. It holds (SI1, SI2, LEN, data) subfields, and BGZF defines one with the id BC carrying a 16-bit BSIZE: the total length of this member, minus one.

if (si1 === 0x42 && si2 === 0x43 && len === 2) {   // "BC"
  bsize = (data[0] | (data[1] << 8)) + 1;          // total member length
}

That single field is the difference between a file you can seek into and one you cannot, and it is why the OmniViewer STATS tab can answer “is this seekable?” at all. When it is present, walking the member chain is pure arithmetic: read 18 bytes, add BSIZE, repeat. When it is absent — the ordinary case — there is no way to find the end of a member except to decode it.

And at the end of every member sit the two fields everything else depends on:

Because the trailer is always the last eight bytes of the file, you can read the final member's checksum and length of a 20 GB archive with a single tiny read. That is what gzip -l does, and it is also why gzip -l lies about anything over 4 GB: ISIZE wraps. A file unpacking to 5 GB honestly reports 1 GB, and no tool reading that field can do better.

Inside the DEFLATE stream

Between the header and the trailer is a DEFLATE stream, and it is a bit stream, read least-significant-bit first. It is a sequence of blocks, each starting with three bits:

BFINAL  1 bit   1 = this is the last block of the member
BTYPE   2 bits  00 stored · 01 fixed Huffman · 10 dynamic Huffman · 11 invalid

Nothing after the first block is byte-aligned. Block two begins wherever block one happened to stop — possibly five bits into a byte — which is why OmniViewer's BLOCKS tab reports offsets in bits. Rounding them to bytes would be a lie about where they are.

The three block types are three different bets the compressor makes:

• Stored (00) — give up. Align to a byte, write a 16-bit length and its ones-complement, then the raw bytes. This is what you get for already-compressed or random data, where entropy coding would make it larger. It is also all you get from gzip -0.
• Fixed Huffman (01) — use the code table baked into the RFC. No table to transmit, so it wins for very short data where writing a custom table would cost more than it saves.
• Dynamic Huffman (10) — build a code table for these bytes specifically and write it out in front of them. Almost everything real is this.

Inside a compressed block, the data is LZ77 followed by Huffman coding. LZ77 replaces a repeat with a back-reference: “copy 12 bytes from 300 bytes ago”. Huffman then gives the common symbols short codes. Concretely, decoding a symbol yields one of three things:

• 0–255 — a literal byte.
• 256 — end of block.
• 257–285 — a match. The symbol picks a base length, some extra bits refine it (3 to 258 bytes), then a second Huffman code gives a distance symbol and its extra bits (1 to 32,768 bytes back).

A dynamic block's header is the part people find surprising: the code tables are themselves Huffman-coded. It declares how many literal/length codes (HLIT), distance codes (HDIST) and code-length codes (HCLEN) it uses, writes the code-length alphabet in a fixed permuted order chosen so the rarely-used entries fall at the end and can be omitted, and then uses that alphabet to encode the lengths of the other two — with run-length escapes (16 repeats the previous length, 17 and 18 emit runs of zeroes) because long stretches of unused symbols are the norm.

A walker that never inflates

Here is the problem the whole toolkit turns on. To find where a member ends — and therefore where its CRC-32 is, and where the next member starts — you must read every symbol in every block, because the block header only says whether a block is the last one, never how long it is.

So decoding the symbols is unavoidable. But reconstructing the output is not, and that is where the memory goes. A normal inflater keeps a 32 KB sliding window (matches point backwards into it) plus an output buffer. Neither is needed to answer “where does this end?”: a match that says copy 258 bytes from 4,000 back contributes 258 to a running total, and then can be forgotten. There is nothing to look back into if you never reproduce the bytes.

That gives a walker whose peak memory is three small Huffman tables and one read window, whatever the file weighs. The reader is the first half — and the trick is that bits() must stay synchronous, or a Huffman loop over a gigabyte would await millions of times:

// A length/distance pair is at most 15 + 5 + 15 + 13 = 48 bits — six bytes.
// So ensuring EIGHT bytes before each symbol means bits() never runs dry
// mid-symbol, and we await roughly once per megabyte instead of once per bit.
function createBitReader(read, size, start = 0, windowSize = 1 << 20) {
  let buf = new Uint8Array(0), bufOff = start, pos = 0;
  let bitBuf = 0, bitCnt = 0, eof = false;

  return {
    get bitOffset() { return (bufOff + pos) * 8 - bitCnt; },

    async ensure(n) {                       // refill the window if it's low
      if (buf.length - pos >= n) return true;
      if (eof) return false;
      const keep = buf.subarray(pos);
      const from = bufOff + pos + keep.length;
      const next = from >= size ? new Uint8Array(0)
                                : await read(from, Math.min(windowSize, size - from));
      if (!next.length) eof = true;
      const merged = new Uint8Array(keep.length + next.length);
      merged.set(keep, 0); merged.set(next, keep.length);
      buf = merged; bufOff += pos; pos = 0;
      return buf.length >= n;
    },

    bits(n) {                               // LSB-first, and synchronous
      while (bitCnt < n) {
        if (pos >= buf.length) return -1;
        bitBuf |= buf[pos++] << bitCnt;
        bitCnt += 8;
      }
      const out = bitBuf & ((1 << n) - 1);
      bitBuf >>>= n; bitCnt -= n;
      return out;
    },

    align() { const drop = bitCnt & 7; bitBuf >>>= drop; bitCnt -= drop; },
  };
}

Huffman decoding is the canonical bucket-by-code-length method from Mark Adler's own puff.c — no lookup table, because building one costs more than it saves for a walker that runs a bounded number of times:

function buildHuffman(lengths, n) {
  const count = new Int32Array(16);
  for (let i = 0; i < n; i++) count[lengths[i]]++;
  count[0] = 0;                                  // unused symbols aren't codes
  const offs = new Int32Array(16);
  for (let len = 1; len < 16; len++) offs[len] = offs[len - 1] + count[len - 1];
  const symbol = new Int32Array(n);
  for (let i = 0; i < n; i++) if (lengths[i]) symbol[offs[lengths[i]]++] = i;
  return { count, symbol };
}

function decodeSymbol(br, huff) {
  let code = 0, first = 0, index = 0;
  for (let len = 1; len <= 15; len++) {
    code |= br.bits(1);
    const count = huff.count[len];
    if (code - count < first) return huff.symbol[index + (code - first)];
    index += count;
    first = (first + count) << 1;
    code <<= 1;
  }
  return -1;
}

And then the body of a block — the loop that does all the work and keeps none of it:

const LENGTH_BASE  = [3,4,5,6,7,8,9,10,11,13,15,17,19,23,27,31,35,43,51,59,
                      67,83,99,115,131,163,195,227,258];
const LENGTH_EXTRA = [0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0];

async function decodeBlockBody(br, lit, dist) {
  let out = 0, literals = 0, matches = 0;
  for (;;) {
    await br.ensure(8);                     // the ONLY await in the hot loop
    const sym = decodeSymbol(br, lit);
    if (sym < 0) throw new Error('invalid literal/length code');
    if (sym < 256) { out++; literals++; continue; }   // a literal byte
    if (sym === 256) break;                          // end of block

    const li     = sym - 257;
    const length = LENGTH_BASE[li] + br.bits(LENGTH_EXTRA[li]);
    const dsym   = decodeSymbol(br, dist);
    br.bits(DIST_EXTRA[dsym]);              // read the distance and DROP it —
    out += length;                          // we never copy, so we never need it
    matches++;
  }
  return { out, literals, matches };
}

The line that matters is the one that throws information away. A real inflater would take that distance, reach back into its window and copy length bytes. We add length to a counter. That is the entire difference between O(window) and O(1), and it is why the BLOCKS tab can run over a file larger than your machine's memory.

It also gives something for free: because out is summed from the symbols rather than measured from produced bytes, it is the true uncompressed size — with none of ISIZE's 4 GB ceiling.

The full member walk

With the walker in hand, the member chain is short. Read a header; either trust BSIZE or walk the bitstream to find the end; read the trailer; repeat while the next two bytes are 1f 8b.

async function walkMembers(read, size) {
  const members = [];
  let offset = 0;

  while (offset + 18 <= size) {                    // 10-byte header + 8-byte trailer
    const head = await read(offset, Math.min(65536, size - offset));
    const h = parseGzipHeader(head, 0);
    if (!h.ok) break;                              // trailing data, not a member

    let deflateEnd, outBytes = null;
    if (h.bsize) {
      deflateEnd = offset + h.bsize - 8;           // BGZF: declared, no decoding
    } else {
      const br = createBitReader(read, size, offset + h.headerLength);
      await br.ensure(1);
      const w = await walkDeflate(br);             // the walker above
      if (w.error) break;
      deflateEnd = w.endByteOffset;
      outBytes = w.outBytes;
    }

    const t  = await read(deflateEnd, 8);
    const dv = new DataView(t.buffer, t.byteOffset, t.byteLength);
    members.push({
      offset, name: h.name, mtime: h.mtime,
      compressed: deflateEnd - (offset + h.headerLength),
      crc: dv.getUint32(0, true),
      isize: dv.getUint32(4, true),
      outBytes: outBytes ?? dv.getUint32(4, true),  // walked size beats ISIZE
    });
    offset = deflateEnd + 8;
  }
  return { members, trailingBytes: size - offset };
}

Two things worth noticing. The walk finds trailing data for free: if offset does not land exactly on the end of the file, whatever is left belongs to no member. Most tools decompress what they can and ignore this silently. And the walk is bounded in OmniViewer — past a compressed-byte budget it stops and says how far it got, with a button to run the whole file, because a tab that hangs for a minute before painting is a worse answer than an honest partial one.

For the overwhelmingly common single-member file, none of this is needed to say something useful: the header is at the front, the CRC-32 and length are the last eight bytes, and both are one small read away at any size.

Why a .tar.gz is slow to browse, and a .zip is not

This is the question the format's design actually answers, and the comparison is the clearest way to see it.

That last row is the reason .tar.gz won on Unix despite everything above it: a source tree full of similar files compresses far better as one stream than as a thousand independent ones. The cost is that the tar's own headers are inside the compressed data, so there is no index to read.

What can be done is to refuse to hold any of it. A tar is a flat run of 512-byte headers, each followed by its file's bytes padded to a 512-byte boundary — so a listing needs the headers and nothing else:

// Stream the payload; read the headers, count past the file contents.
let skip = 0, offset = 0, zeroBlocks = 0;

for await (const chunk of decompressed(file)) {   // DecompressionStream('gzip')
  let p = 0;
  while (p < chunk.length) {
    if (skip > 0) {                    // inside a file's bytes: don't keep them
      const take = Math.min(skip, chunk.length - p);
      skip -= take; p += take; offset += take;
      continue;
    }
    const block = take512(chunk, p);   // straddles chunks; buffer the remainder
    const h = parseTarHeader(block);
    if (h === null) {                  // two zero blocks end the archive
      if (++zeroBlocks >= 2) return entries;
      continue;
    }
    zeroBlocks = 0;
    entries.push({ name: h.name, size: h.size, mode: h.mode, mtime: h.mtime });
    skip = Math.ceil(h.size / 512) * 512;         // the payload, rounded up
  }
}

Peak memory is one chunk plus at most 511 buffered bytes of a straddling header. Listing a 20 GB .tar.gz costs time proportional to its size — unavoidably — but memory proportional to nothing at all.

The check that DEFLATE doesn't have

The single most useful thing to know about this format: DEFLATE has no integrity check inside it.

We measured it. Take a small gzip member, flip one bit at a time through every byte of its compressed data, and run the walker over each of the 64 results:

In four cases out of five a single-bit corruption produces a stream that decodes perfectly — into different bytes than were put in. No error, no warning. This is not a bug in DEFLATE; it is what a pure entropy coder is. Almost every bit pattern is a legal symbol sequence.

Which is exactly why gzip appends a CRC-32 and a length to every member, and why checking them means decompressing everything. There is no shortcut. The only concession is that you need not keep anything:

// Verify one member: inflate its byte range, fold, discard.
let state = crc32Init(), out = 0;
await streamInflate('deflate-raw', m.deflateStart, m.deflateEnd - m.deflateStart,
  (chunk) => { state = crc32Update(state, chunk); out += chunk.length; });

const crcOk  = crc32Final(state) === m.crc;
const sizeOk = (out >>> 0) === m.isize;   // ISIZE is mod 2^32 — compare the same way

Note deflate-raw rather than gzip: the header has already been parsed, so what is handed to the platform decompressor is the bare bitstream over one member's exact byte range. That is what makes it a per-member check rather than a single pass over the file, and it means a member far larger than memory still verifies — the cost is time and reading, never RAM. It is also why no compression library ships with OmniViewer at all: DecompressionStream is in every modern browser, and the walker above needs no inflater.

Limits and caveats

• ISIZE wraps at 4 GB. The field is 32 bits, so a file unpacking to 5 GB reports 1 GB — gzip -l included. Summing the walker's symbol counts has no such ceiling, which is why the whole-file walk gives a different (and correct) number.
• Walking is not free. Decoding symbols to find a member's end costs roughly what inflating costs, minus the copying. The automatic walk is therefore budgeted, and the whole-file pass is a button rather than a default.
• A clean walk proves nothing about integrity. See above. Only VERIFY answers that question.
• There is still no random access. Nothing here makes an ordinary gzip file seekable; it makes reading it bounded in memory, which is a different and lesser claim. If you need to seek, you need BGZF — the choice has to be made at compression time.
• Multi-member files confuse many tools. Plenty of libraries stop after the first member and silently return a fraction of the data. The member chain is the first thing to check when a decompressed file looks truncated.

OmniViewer opens every file format in your browser — ZIP, JSON, CSV, Parquet, PDF and more — powered by the same windowed engine as fastjsonviewer.com and hugecsv.com. Gzip is one of the formats with dedicated tooling.