How a PDF works: objects, the cross-reference table & streams

A PDF is not a stream of pages. It is a tree of numbered objects with an index at the end that records where each one lives — which is exactly why you can open a 20 GB PDF instantly, seek to a single page, and rewrite the file to strip its metadata without touching the rest. This is a tour of the format, with real JavaScript you can lift.

What is a PDF?

PDF — Portable Document Format — was created by Adobe in 1993 and has been an open ISO standard (ISO 32000) since 2008. Its promise is that a document looks the same everywhere, so it carries its own fonts, its own layout coordinates, and its own imaging model. Underneath that promise is a surprisingly simple object database, described in a mix of ASCII keywords and binary bytes.

A PDF file has four parts, in this order: a one-line header (%PDF-1.7), a body of numbered objects, a cross-reference table that indexes them, and a trailer that says where to start. The trick to the whole format is that you read it back to front.

The eight object types

Every value in a PDF is one of eight types. If you have ever read JSON, six of them are familiar:

The two that make PDF a database rather than a document are the indirect object and the indirect reference. Any object can be given a number and a generation and defined at the top level:

12 0 obj
<< /Title (Quarterly report) /Author (A. Person) >>
endobj

and referred to from anywhere else by 12 0 R. So the trailer does not contain the document info — it points at it: /Info 12 0 R. Objects reference objects, and the file is that graph, flattened.

A whole tiny PDF

Here is a complete, valid one-page PDF. Four objects: the catalog (the root), the page tree, one page, and the page’s content stream — the drawing instructions.

%PDF-1.7
1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj
2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 >> endobj
3 0 obj << /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792]
          /Resources << /Font << /F1 5 0 R >> >> /Contents 4 0 R >> endobj
4 0 obj << /Length 44 >>
stream
BT /F1 24 Tf 72 720 Td (Hello, PDF) Tj ET
endstream
endobj
5 0 obj << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >> endobj

Read the graph from the root: catalog → pages → page → content. The content stream draws text with the operators BT (begin text), Tf (set font), Td (move), Tj (show string) and ET. Everything you see on a PDF page is a sequence like this.

The cross-reference table

How does a reader find object 3’s bytes without scanning the file? The cross-reference table — the xref — is a directory of byte offsets, one line per object, each line exactly 20 bytes so a reader can jump to entry n by arithmetic:

xref
0 6
0000000000 65535 f
0000000009 00000 n
0000000058 00000 n
0000000115 00000 n
0000000241 00000 n
0000000335 00000 n
trailer
<< /Size 6 /Root 1 0 R /Info 12 0 R >>
startxref
412
%%EOF

Each n entry is <10-digit offset> <5-digit generation> n. Object 4 lives at byte 241. The very last thing in the file is startxref followed by the byte offset of the xref keyword itself. That is why you read a PDF backwards: seek to the end, read startxref, jump to the table, and now you have the offset of every object.

// Find the cross-reference: the last `startxref` in the file tail.
function findStartXref(tail) {
  const i = tail.lastIndexOf('startxref');
  return parseInt(tail.slice(i + 9), 10);   // → byte offset of the xref
}

When a PDF is edited, the writer can append rather than rewrite: new objects, a small new xref section, and a trailer whose /Prev points at the previous table. A reader chains through /Prev, newest first, so a document can be saved a hundred times and still open in one seek. The number of those sections is the file’s edit history.

Streams & filters

The bytes between stream and endstream are almost always compressed. The stream dictionary’s /Filter names the codec; by far the most common is FlateDecode — zlib/DEFLATE, the same algorithm as a .zip or PNG. Modern browsers ship a DEFLATE decoder, so decoding a PDF stream needs no library at all:

async function inflate(bytes) {
  const ds = new DecompressionStream('deflate');           // zlib wrapper
  const stream = new Blob([bytes]).stream().pipeThrough(ds);
  return new Uint8Array(await new Response(stream).arrayBuffer());
}

Other filters you meet: LZWDecode (older), ASCIIHexDecode and ASCII85Decode (text-safe wrappers), RunLengthDecode, and the image codecs DCTDecode (JPEG) and JPXDecode (JPEG 2000). A stream can even chain filters. Flate streams are often paired with a predictor — the same PNG row filters (Sub, Up, Average, Paeth) that make similar rows compress better — which you undo after inflating.

Cross-reference streams & object streams

PDF 1.5 (2003) added two things that break naïve parsers, so a real reader must handle them. First, the xref table can itself be a cross-reference stream: a compressed binary object (/Type /XRef) instead of the ASCII table above. Its /W [1 4 2] gives the byte widths of three fields per entry — type, field 2, field 3 — and the whole thing is Flate-compressed with a predictor:

// One entry = type(w0) + f2(w1) + f3(w2) bytes, big-endian.
//   type 1 → f2 is a byte offset, f3 the generation (a normal object)
//   type 2 → f2 is the number of the object stream that holds it
//   type 0 → a free entry
for (let p = 0; p + rec <= data.length; p += rec) {
  const type = readBE(data, p, w0);
  const f2   = readBE(data, p + w0, w1);
  const f3   = readBE(data, p + w0 + w1, w2);
  // …record the entry for this object number…
}

Second, many objects are no longer in the file directly — they are packed, several at a time, inside an object stream (/Type /ObjStm): a Flate-compressed blob whose header lists object number → offset, then the objects concatenated. This is why a modern PDF’s catalog and metadata may be invisible to grep: they are compressed inside another object. A type-2 xref entry says “object 6 lives at index 1 of object stream 30”, and the reader inflates stream 30 to pull it out.

Index-then-seek: reading a 20 GB PDF

Put the pieces together and the scale story falls out. Because the index is at the tail and records a byte offset for every object, you never need the whole file in memory. To open a document of any size:

1. Read a bounded window at the tail — the last few megabytes — to find startxref and parse the cross-reference (and its /Prev chain).
2. Read a bounded window at the head — the header and the first objects and pages.
3. Seek to any other object on demand, using the offset the xref gave you.

An object whose offset falls in the un-loaded middle is simply resolved lazily — or, for a genuinely enormous file, reported as “beyond the loaded window” rather than forcing a multi-gigabyte read. The catalog, the info dictionary and the page tree almost always live near the head or the tail (they are written last), so the document opens on a few megabytes of reads no matter how big it is. OmniViewer’s raw, hex and strings views stay windowed over every byte on top of that.

Getting the text back out

Extracting text is not “read the strings” — the words live inside content streams as show operators, drawn through a font. You decode each page’s content stream and watch for Tj (show one string), TJ (show an array of strings with kerning adjustments), and the line-movement operators to reconstruct spaces and newlines:

// A content-stream fragment and what it means:
BT                     % begin text
/F1 12 Tf              % font F1 at 12 pt
72 720 Td (The quick) Tj   % move, show "The quick"
0 -14 Td (brown fox) Tj    % next line, show "brown fox"
ET                     % end text

The bytes in each string are glyph codes, not Unicode. If the font carries a /ToUnicode CMap, you map codes → characters through it; otherwise you fall back to the font’s encoding (usually WinAnsi, i.e. Windows-1252). Text drawn as vector outlines, or a page that is just a scanned image, has no operators to decode — that text was never really text, which is why it can’t be selected in any viewer.

Cleaning metadata by rewriting, not appending

A PDF quietly records who made it and when: the trailer’s /Info dictionary (Author, Producer, CreationDate, ModDate) and the catalog’s XMP metadata stream. You could “remove” them the way PDF editors do — append an incremental update that overrides the trailer — but the original bytes would still sit in the file, one strings command away. For privacy that is worthless.

The honest fix is a rewrite. Because a PDF is just a graph of objects, you can walk every object you want to keep, renumber them densely, and emit a fresh file with a brand-new cross-reference table and a trailer that has no /Info — dropping the info dictionary and every XMP stream entirely. Page content streams are copied verbatim (their already-compressed bytes), so nothing is re-encoded and the document is pixel-identical:

// Keep every object except the Info dict and any Metadata/XRef/ObjStm.
const kept = [...xref.keys()].filter((n) => n !== infoNum && !isMetadata(n));
kept.forEach((n, i) => remap.set(n, i + 1));      // dense renumber, gen 0

// Re-emit: header, each object (refs remapped), fresh xref, trailer w/o /Info.
// Streams keep their raw bytes → no re-encode, pages unchanged.

The result: the author, producer, dates and editing software are gone from the actual bytes, verifiably absent from a search of the cleaned file — while the pages, the fonts and the text are untouched. That is the difference between hiding metadata and removing it.