Inside a ZIP file: the central directory and DEFLATE

A ZIP file keeps its table of contents at the end. That one decision — made in 1989, for floppy disks — is why you can list every file in a 20 GB archive with two reads and no unpacking, why a ZIP can be glued onto the back of an installer, and why the format has exactly the security corners it has. Here is the whole layout, with real code.

What a ZIP is

Phil Katz published the ZIP format in 1989, along with PKZIP, and — unusually for the era — put the format specification in the public domain. That is the entire reason ZIP won. The spec is still maintained as PKWARE’s APPNOTE.TXT, and it is still readable in an afternoon.

What most people don’t realise is how much of the software world is literally a ZIP file. All of these are ZIP archives with a different extension and some required members inside:

So “open a ZIP” and “look inside a .docx” are the same operation. OmniViewer routes every one of them to the same toolkit, and it detects them by their first four bytes — 50 4B 03 04, which is PK followed by 3 and 4, Phil Katz’s initials and a version number — not by the extension.

The layout: members first, index last

A ZIP is three regions, in this order:

┌─────────────────────────────────────────────────────────┐
│ MEMBER AREA                                             │
│   [local header][compressed bytes]   ← entry 1          │
│   [local header][compressed bytes]   ← entry 2          │
│   …                                                     │
├─────────────────────────────────────────────────────────┤
│ CENTRAL DIRECTORY                                       │
│   [central header]  ← entry 1: name, sizes, CRC, offset  │
│   [central header]  ← entry 2                            │
│   …                                                     │
├─────────────────────────────────────────────────────────┤
│ END OF CENTRAL DIRECTORY (22 bytes + comment)            │
│   how many entries · directory size · directory offset   │
└─────────────────────────────────────────────────────────┘

Every member is stored independently: its own header, its own compression, its own CRC. There is no shared dictionary across members and no global stream. That independence is the format’s superpower — you can decompress entry 900 without touching entries 1–899 — and also why a ZIP compresses worse than a .tar.gz, which gzips the whole concatenated stream and so gets to reuse patterns across files.

Why is the index at the end? Because in 1989 you wrote an archive to a floppy sequentially, and you did not know the compressed size of a file until you had compressed it. Writing the directory last means a ZIP can be produced in a single forward pass, streaming, with no seeking back — and, as a bonus, appending to an archive means rewriting only the tail.

Finding the end of the file

To read anything, you first find the End of Central Directory record (EOCD). It is 22 bytes and its signature is 50 4B 05 06:

The annoyance is that trailing comment: the record can sit up to 65,535 bytes from the end of the file. So you read a tail window and scan backwards for the signature — backwards, because a comment (or a compressed payload, or a nested ZIP) can contain the bytes PK\x05\x06 by chance, and the last match is the real one.

const EOCD_SIG = 0x06054b50;
const EOCD_MIN = 22;
// 65535 (max comment) + the record itself + the two ZIP64 records that may
// precede it. One read of this size always covers the end of any valid archive.
const TAIL = 65535 + EOCD_MIN + 56 + 20;

const u16 = (b, p) => b[p] | (b[p + 1] << 8);
const u32 = (b, p) => (b[p] | (b[p + 1] << 8) | (b[p + 2] << 16)) + b[p + 3] * 0x1000000;

async function findEocd(file) {
  const start = Math.max(0, file.size - TAIL);
  const tail = new Uint8Array(await file.slice(start).arrayBuffer());

  for (let p = tail.length - EOCD_MIN; p >= 0; p--) {
    if (u32(tail, p) !== EOCD_SIG) continue;
    const commentLen = u16(tail, p + 20);
    // Consistency check: a real EOCD's comment ends at or before EOF. This is
    // what rejects a signature that is really payload.
    if (start + p + EOCD_MIN + commentLen > file.size) continue;
    return {
      offset: start + p,
      totalEntries: u16(tail, p + 10),
      cdSize: u32(tail, p + 12),
      cdOffset: u32(tail, p + 16),
      trailing: file.size - (start + p + EOCD_MIN + commentLen),
    };
  }
  return null;   // truncated download, or not a ZIP
}

The central directory

The EOCD gives you an offset and a size; read that range and you have every member’s metadata. Each record is 46 fixed bytes plus three variable-length blocks, signature 50 4B 01 02:

Walking it is a straight loop — no recursion, no lookahead:

function parseCentralDirectory(cd) {
  const entries = [];
  let p = 0;
  while (p + 46 <= cd.length && u32(cd, p) === 0x02014b50) {
    const flags    = u16(cd, p + 8);
    const nameLen  = u16(cd, p + 28);
    const extraLen = u16(cd, p + 30);
    const cmtLen   = u16(cd, p + 32);
    const name = new TextDecoder(flags & 0x800 ? 'utf-8' : 'ibm437')
      .decode(cd.subarray(p + 46, p + 46 + nameLen));

    entries.push({
      name,
      isDirectory: name.endsWith('/'),
      method: u16(cd, p + 10),
      crc32: u32(cd, p + 16),
      compressedSize: u32(cd, p + 20),
      uncompressedSize: u32(cd, p + 24),
      localHeaderOffset: u32(cd, p + 42),
      unixMode: (u32(cd, p + 38) >>> 16) & 0xffff,
      encrypted: (flags & 1) !== 0,
    });
    p += 46 + nameLen + extraLen + cmtLen;
  }
  return entries;
}

Three details that trip people up:

• The name is a path, and it is not sanitised. The spec says use forward slashes and relative paths. Nothing enforces it. This is the root of zip slip.
• Encoding. Flag bit 11 means the name is UTF-8. Without it, the name is historically CP437, the IBM PC OEM code page. Decoding an old archive as UTF-8 is how you get mojibake filenames.
• There are no folders. A directory entry is just a zero-length member whose name ends in /, and it is optional. Plenty of archives contain src/lib/util.js with no entry for src/ or src/lib/ at all — so a viewer has to build the tree from path segments, which is what OmniViewer’s ENTRIES tab does.

ZIP64: what happens past 4 GB

Every size and offset above is 32 bits. In 1989 that was absurd headroom; today it is a 4 GB ceiling, and the entry count ceiling is 65,535. ZIP64 (added in 2001, APPNOTE 4.5) fixes this without breaking old readers, using a trick worth knowing:

When a value doesn’t fit, the classic field is written as all ones — 0xFFFFFFFF, or 0xFFFF for the 16-bit counts — and the real 64-bit value goes elsewhere. For a per-entry value, “elsewhere” is a ZIP64 extended information extra field (header ID 0x0001). For the archive-wide values, it is a second EOCD:

[ZIP64 end of central directory record]   ← 56 bytes, sig 50 4B 06 06
[ZIP64 end of central directory locator]  ← 20 bytes, sig 50 4B 06 07
[end of central directory]                ← 22 bytes, sig 50 4B 05 06  ← still last

So you always find the classic EOCD first, then look 20 bytes before it for the locator, then follow the locator to the real record. The one genuinely error-prone part is the extra field, because it is positional: it contains only the values whose classic field was saturated, in a fixed order, with no tags.

// The ZIP64 extra field (0x0001) holds, IN THIS ORDER, only the fields whose
// classic value was 0xFFFFFFFF. Read blindly and every offset after the first
// missing value is wrong — the classic ZIP64 parsing bug.
function parseZip64Extra(body, need) {
  const out = {};
  let q = 0;
  if (need.uncompressed && q + 8 <= body.length) { out.uncompressedSize   = u64(body, q); q += 8; }
  if (need.compressed   && q + 8 <= body.length) { out.compressedSize     = u64(body, q); q += 8; }
  if (need.offset       && q + 8 <= body.length) { out.localHeaderOffset  = u64(body, q); q += 8; }
  if (need.disk         && q + 4 <= body.length) { out.diskStart          = u32(body, q); q += 4; }
  return out;
}

The extra-field block is a general mechanism — [id: 2][size: 2][body: size], repeated — and it is where the rest of the modern metadata lives: high-resolution timestamps with an actual timezone (0x5455 Info-ZIP, 0x000A NTFS), unix uid/gid (0x7875), and WinZip AES encryption (0x9901, which also hides the real compression method, because the method field reads 99 for every AES entry).

Every entry is described twice

Before each member’s compressed bytes sits a local file header (signature 50 4B 03 04) carrying almost the same fields as the central directory record: name, method, CRC, both sizes. The format deliberately stores the metadata twice.

There are good reasons. A streaming extractor that reads the file front-to-back needs the metadata before the data. And if the directory at the end is destroyed, the local headers are enough to recover the archive by brute-force scanning.

But it means the two copies can disagree, and the spec does not require any tool to check. Since almost every tool lists from the central directory and some extract using the local header, a deliberate mismatch shows a scanner one file and an extractor another. This is a real, repeatedly exploited technique — most famously against Android, where the Master Key bug (2013) and later Janus exploited exactly this disagreement to keep a valid APK signature while swapping the code.

// Cross-check one entry. Bit 3 (data descriptor) means the local header
// legitimately writes zeros and the real sizes trail the data — so those
// fields are not evidence of anything.
function compareHeaders(central, local) {
  const bad = [];
  if (local.name !== central.name) bad.push('name');
  if (local.method !== central.method) bad.push('method');
  if (!local.hasDataDescriptor) {
    if (local.crc32 !== central.crc32) bad.push('crc');
    if (local.compressedSize !== central.compressedSize) bad.push('compressed size');
    if (local.uncompressedSize !== central.uncompressedSize) bad.push('size');
  }
  return bad;
}

One more consequence: the local header has its own name and extra-field lengths, which can differ from the central copy. So the first compressed byte is at localHeaderOffset + 30 + localNameLen + localExtraLen — computed from the local header, never assumed from the directory.

DEFLATE, in one screen

Method 8, DEFLATE (RFC 1951, also by Phil Katz), is what over 95% of real-world ZIP entries use. It is two ideas stacked:

1. LZ77: replace a repeated sequence with a back-reference — “go back 214 bytes and copy 9”. The window is 32 KB, which is why compression only finds repetition that is reasonably close together.
2. Huffman coding: encode the resulting stream of literals, lengths and distances with variable-length bit codes, so common symbols cost fewer bits. The code tables are either a fixed set defined in the RFC or written into the block itself.

That is also the whole explanation for compression ratios. A file of one repeated byte is nothing but back-references, so it compresses about 1030:1 — the theoretical maximum for a deflate stream. Already-compressed data (a PNG, a JPEG, an MP4) has no exploitable repetition left, so deflating it gains nothing and costs CPU; a good ZIP writer notices and stores those members instead. When you see method stored next to the images in an archive and deflate next to the text, that is a well-built archive.

Extracting exactly one entry

Everything above adds up to this: pulling one file out of an archive is four small reads and no unpacking of anything else.

async function extractEntry(file, entry) {
  // 1. Read the local header (its lengths decide where the data starts).
  const head = new Uint8Array(await file.slice(entry.localHeaderOffset,
                                              entry.localHeaderOffset + 4096).arrayBuffer());
  if (u32(head, 0) !== 0x04034b50) throw new Error('no local header here');
  const dataStart = entry.localHeaderOffset + 30 + u16(head, 26) + u16(head, 28);

  // 2. Read ONLY this entry's compressed bytes.
  const packed = new Uint8Array(
    await file.slice(dataStart, dataStart + entry.compressedSize).arrayBuffer());

  // 3. Inflate with the platform decompressor (stored entries are a copy).
  let bytes = packed;
  if (entry.method === 8) {
    const ds = new DecompressionStream('deflate-raw');
    const writer = ds.writable.getWriter();
    writer.write(packed);            // not awaited: the reader below drains it
    writer.close();
    bytes = new Uint8Array(await new Response(ds.readable).arrayBuffer());
  }

  // 4. Verify the archive's own promise about the contents.
  if (crc32(bytes) !== entry.crc32) throw new Error('CRC-32 mismatch — corrupt entry');
  return bytes;
}

The CRC-32 check is the step people skip, and it is the one that turns “it decompressed” into “it is intact”. ZIP mandates the same polynomial PNG uses (0xEDB88320, reflected), and the table is eight lines of code.

Reading a 20 GB archive

Here is the payoff of the index-at-the-end design. To show the complete contents of an archive, OmniViewer reads:

1. a tail window (~64 KB) — find the EOCD, and the ZIP64 records if present;
2. the central directory range it names — about 46 bytes plus the path per entry, so ~10 MB for a quarter of a million files;
3. a bounded sample of local headers (~4 KB each, a few dozen) to cross-check them against the directory.

That is it. The compressed payloads — which is to say essentially the entire file — are never read. The cost of listing an archive scales with its number of entries, not its size, so a 20 GB release archive opens as fast as a 20 KB one. A single entry is only decompressed when you ask for that entry, and then only its own byte range is fetched.

In a browser this matters more than it sounds, because a File from a drop or a file picker is a handle, not bytes: file.slice(a, b) is a lazy range request against the disk. Nothing is loaded until you await it. The same code path runs against a URL using HTTP Range requests, so an archive can be inspected without downloading it at all.

The dangerous parts

ZIP’s security problems are not implementation bugs; they are consequences of the design above. All of them are visible in the index, which means all of them can be detected before you extract anything. That is what the AUDIT tab does.

Zip slip (path traversal)

An entry name is an arbitrary byte string that extractors treat as a path. Name a member ../../../../etc/cron.d/backdoor and a naive extractor — one that joins the output directory with the entry name and writes — writes outside the output directory. Absolute names (/etc/passwd), drive-qualified names (C:\Windows\...) and backslash separators do the same job; backslashes are especially effective because a sanitiser that only splits on / never sees the traversal.

// Reject, do not sanitise. Normalising and hoping is how these get through.
const unsafe = (name) =>
  /(^|[/\\])\.\.([/\\]|$)/.test(name) ||   // any ".." segment
  name.startsWith('/') ||                  // absolute (POSIX)
  /^[A-Za-z]:[/\\]/.test(name) ||          // drive-qualified (Windows)
  name.startsWith('\\\\');                 // UNC path

The related case is symlinks. A symlink is stored as a tiny member whose contents are the target path and whose unix mode has S_IFLNK set. Extract a link pointing at /etc, then extract a second member “into” that link, and the write lands outside the directory without any entry name ever containing ...

Zip bombs

Because the directory declares the uncompressed size, you can see an expansion before paying for it. The classic 42.zip nests archives inside archives to reach petabytes; the modern non-recursive version (Fifield, 2019) overlaps a single deflate stream across many central-directory entries so one physical payload is decompressed hundreds of times over. Both are obvious in the index:

• a single entry expanding more than a few hundred times — ordinary data never does, a run of one byte does;
• a whole archive whose declared uncompressed total dwarfs its size on disk;
• many entries whose local header offsets point at the same place — the overlap trick.

Ratios only mean something above a size floor, incidentally: on a 40-byte file the fixed deflate overhead dominates and the “ratio” is noise.

Header disagreement, and duplicate names

Covered above: the two metadata copies can differ, and different tools trust different copies. A close relative is the duplicate entry name — nothing forbids two members called config.json, and which one ends up on disk is down to the extractor’s ordering, so one can quietly shadow the copy a reviewer inspected.

Encryption

Two schemes exist. Legacy ZipCrypto is broken — a known-plaintext attack recovers the key, and knowing a few bytes of one member is enough. WinZip AES (the 0x9901 extra field) is real AES-128/192/256. Either way the entry names and sizes stay in the clear: a ZIP encrypts contents, never its table of contents. That is why an encrypted archive still tells you what is in it.

Try it on a real archive

Everything described here runs locally in your browser — no upload, no server, and only the bytes needed are ever read.

References

• PKWARE, APPNOTE.TXT — the ZIP File Format Specification (the normative source; ZIP64 is §4.5, encryption §7).
• RFC 1951 — DEFLATE Compressed Data Format Specification, version 1.3.
• RFC 1952 — GZIP file format (the wrapper DEFLATE usually wears elsewhere).
• Info-ZIP, proginfo/extrafld.txt — the extra-field IDs the spec leaves to convention.
• David Fifield, A better zip bomb (2019) — the non-recursive overlapping-stream construction.
• Snyk, Zip Slip (2018) — the path-traversal class and the affected libraries.