Extract text from a PDF: why copy-paste breaks, and how to fix it

Someone sent you a PDF and you need the words out of it — to quote it, search it, diff two versions, or paste it somewhere that isn’t a PDF. Select-all and copy gives you thewordsallrunt ogether, or a wall with no paragraphs, or nothing at all. That is not your reader being lazy: a PDF contains no spaces and no lines. It contains a pen. Here is what it takes to get real text back, and the JavaScript that does it.

What you actually get

Drop a PDF on omniviewer.org/pdf and the TEXT tab hands you the document’s words, page by page, with the line breaks and paragraphs put back — ready to copy, or to download as a .txt. If the file is a scan, it tells you that instead of showing you an empty box. If the fonts are broken and the “text” is mojibake, it warns you rather than letting the garbage pass as a successful extraction.

All of it runs in your browser, in a Web Worker, on a static page. The contract you were sent, the medical report, the paper under embargo — none of it is uploaded anywhere, because there is nowhere to upload it to.

Here is the head of the “Attention Is All You Need” paper coming back out of the extractor, verbatim (from the middle of page 1) — author names, affiliations and emails each intact on their own line, paragraph breaks where the paper has them:

Attention Is All You Need

Ashish Vaswani∗ Noam Shazeer∗ Niki Parmar∗ Jakob Uszkoreit∗
Google Brain Google Brain Google Research Google Research
avaswani@google.com noam@google.com nikip@google.com usz@google.com

  […the other three authors, same shape…]

Abstract

The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks…

15 pages, 39,734 characters, three different font-encoding mechanisms in one document, zero fonts that failed to map. Getting to that output is the rest of this article.

A PDF has a pen, not words

The text you see on a page lives inside that page’s content stream: a little program of drawing operators, usually Flate-compressed. There is no paragraph, no line, no word and no space in it — there is a current point and instructions to show glyphs there. A fragment:

BT                     % begin text
/F1 12 Tf              % font F1 at 12 pt
72 720 Td (The quick) Tj   % move the pen, show "The quick"
0 -14 Td (brown fox) Tj    % move down 14 pt, show "brown fox"
ET                     % end text

Two facts follow, and they explain nearly every bad paste you have ever had:

• A new line is a pen movement (Td, T*, Tm), not a newline character. Read the strings in order and you get one endless line.
• A space is usually a gap, not a character. A producer may draw every word — sometimes every glyph — as its own string at its own coordinate. Concatenate them and the words fuse.

So “extracting the text” is not string collection. It is running the page’s program, tracking where the pen ends up, and reading the geometry back as language.

The interpreter: a 200-line stack machine

The content stream is postfix: operands are pushed, an operator consumes them. So the extractor is a tokenizer plus a switch over the operators that move the pen or change the text state. The whole text-relevant instruction set is small:

Showing a string does two things: it emits a positioned item, and it advances the pen by exactly what the glyphs measure. That sum is straight out of PDF 32000-1 §9.4.4, and it is the heart of the whole exercise:

// Show one string: read it through the font, emit an item where the pen is,
// and move the pen on by what the glyphs measured.
const show = (val) => {
  const m = (font || FALLBACK).measure(val.bytes);
  // Each glyph displaces the pen by its own width (thousandths of an em)
  // scaled by the font size, plus character spacing, plus word spacing on a
  // literal code 32 — the whole sum multiplied by the horizontal scale.
  const tx = ((m.w / 1000) * size + charSp * m.count + wordSp * m.spaces) * hscale;
  const before = textMatrix();
  tm = mul([1, 0, 0, 1, tx, 0], tm);
  if (!m.text) return;                    // unreadable glyphs still move the pen
  const after = textMatrix();
  items.push({
    x: before[4], y: before[5], w: after[4] - before[4],
    // The *effective* size on the page: a 1 pt font in a 12× matrix is 12 pt text.
    size: Math.hypot(before[2], before[3]) || size,
    text: m.text,
  });
};

Two details in there matter more than they look. The pen advances even for text we cannot read — a subset font with no character identity still occupies its width, and skipping it would shift every subsequent word on the line. And the item’s recorded size comes from the composed matrix, not from the Tf operand: the font size in a PDF is a multiplier, and a producer that sets 1 Tf under a 12× transform is drawing 12 pt text. Get that wrong and every threshold downstream — space widths, line grouping — is wrong by the same factor.

The position where the pen actually is on the page is three matrices composed: the font’s own scale, the text matrix, and everything cm and the enclosing forms did to user space.

const textMatrix = () => mul([size * hscale, 0, 0, size, 0, rise], mul(tm, ctm));

Codes are not characters

A string in a content stream is a sequence of character codes, and a code means nothing until you know the font that draws it. (\x48\x69) is “Hi” in one font and two arbitrary glyph indices in another. Working out the mapping is the other half of extraction, and it has a fallback chain:

1. /ToUnicode — the producer wrote down code → Unicode as a CMap. Parse the bfchar / bfrange entries and you are done. Most modern producers supply this.
2. Base encoding + /Differences — a simple font names an encoding (Standard, WinAnsi, MacRoman) and may re-point individual codes at glyph names. A glyph name is resolvable: eacute is é, and uniXXXX spells the code point outright. Build a 256-entry table, then patch it.
3. Predefined or embedded CMaps — a composite (Type0) font names a CMap. The name alone tells you a lot: the …-UCS2-H family means the code is the Unicode value, and the legacy CJK families (RKSJ = Shift_JIS, GBK-EUC, ETen-B5, KSC-EUC) name character sets the platform’s own TextDecoder already knows. An embedded CMap stream additionally gives the codespace ranges — how many bytes each code takes — which is what stops a multi-byte string being sliced into nonsense.
4. Nothing. A subset font with Identity ordering, no /ToUnicode and glyph names like g17 carries glyph indices into an embedded font program. Those codes have no character identity at all. There is no clever trick here — only OCR, or the honest admission.

That last case is why so many “PDF to text” tools emit mojibake: they map the code through Latin-1 anyway and ship the result. The extractor here marks the decoder mapped: false, counts those fonts, and says which pages they cost you. The sample paper reports fontKinds: ['tounicode', 'builtin', 'encoding'] — three mechanisms in one 15-page document — and unmappedFonts: 0.

Decoders are built once per font object, not once per page: a 500-page report typically uses a handful of fonts, each /ToUnicode parse is real work, and the cache is keyed by object number so a font shared across pages is built once.

Where the spaces come from

Now the interesting part. Given positioned items with real widths, a space is not a character to look for — it is a measurement. Three ways the same gap gets drawn:

[(W) 120 (alk)] TJ         % kern -0.12 em: still one word, "Walk"
[(Walk) -400 (on)] TJ      % 0.4 em of nothing: that is a space
72 700 Td (Left) Tj  272 700 Td (Right) Tj   % 138 pt apart: also a space

A number inside a TJ array moves the pen backwards by that many thousandths of an em. Treating it as a real displacement — rather than ignoring it, or blindly turning every one into a space — is what lets tight letter-spacing be told apart from a word break:

case 'TJ': {
  for (const el of top) {
    if (typeof el === 'number') tm = mul([1, 0, 0, 1, (-el / 1000) * size * hscale, 0], tm);
    else if (el instanceof PdfString) show(el);
  }
  break;
}

And then the rule that turns geometry into language is one line: insert a space when the gap exceeds a fifth of the type size (SPACE_EM = 0.2). Below that it is kerning; above it, the pen visibly jumped.

Lines and paragraphs from baselines

Lines come back the same way. Items are sorted down the page and then across it (-y first, because PDF user space has y increasing upwards), grouped into lines by baseline, ordered left to right, and joined with the space rule above:

const sorted = items.slice().sort((a, b) => (b.y - a.y) || (a.x - b.x));

// Group by baseline. The tolerance grows with the type size, because that is
// what a superscript is: a footnote marker on 10 pt text sits a third of an em
// above the baseline, and belongs to the line it annotates — not a line of its own.
for (const it of sorted) {
  const size = Math.max(line ? line.size : 0, it.size || 0);
  if (!line || line.y - it.y > Math.max(yTol, 0.4 * size)) {
    line = { y: it.y, size: it.size || 0, items: [it] };
    lines.push(line);
  } else { line.size = size; line.items.push(it); }
}

Then, within a line:

for (const it of ln.items) {
  if (pen != null && it.x - pen > Math.max(spaceEm * (it.size || 10), 0.5)
      && !/\s$/.test(text) && !/^\s/.test(it.text)) text += ' ';
  text += it.text;
  // Items can overlap (text drawn twice for a faux-bold effect): the pen is the
  // rightmost edge reached, not the last one written.
  pen = Math.max(pen == null ? -Infinity : pen, it.x + it.w);
}

Paragraphs are the last piece, and they cannot use a fixed threshold: 16 pt of leading is a paragraph break in a footnote and a normal line gap in a title. So the break is relative to this page’s own habits — take the median gap between consecutive baselines, and call anything more than 1.6× that a paragraph break. The median, not the mean, so one figure caption sitting alone at the bottom of the page doesn’t move the threshold for the whole page.

Text hiding inside form XObjects

Not all of a page’s text is in the page’s content stream. The Do operator draws an XObject, and a /Subtype /Form XObject is a reusable content stream with its own /Resources — its own fonts, which are generally not the page’s. Running headers, footers, letterhead, and from some LaTeX and Word exporters entire pages arrive that way. An extractor that only reads the page stream silently loses them.

So the interpreter recurses, composing the form’s /Matrix onto the CTM in force where it was drawn, so its text lands where it is actually painted — a header inside a form sorts to the top of the page like any other text. Two guards keep that safe:

// Cycle guard over the *current path* rather than a seen-once set: a form
// legitimately appears many times on a page (a running header), and each
// appearance is text; only a form that reaches itself must be cut.
if (ctx.visited.has(key)) return;
ctx.visited.add(key);
try {
  const res = (await doc.deref(obj.dict.get('Resources'))) ?? resources; // forms may inherit
  await runContent(bytes, await loadFontsFromResources(doc, res, ctx.cache),
                   { ...ctx, resources: res, depth: ctx.depth + 1 }, mul(formMatrix, ctm));
} finally {
  ctx.visited.delete(key);   // pop: the same form may be drawn again later
}

The distinction between a path-scoped guard and a global “seen” set is the difference between extracting a running header once per page and extracting it exactly once in the whole document. Depth is capped at 8 and total form runs at 4,000 per page, so a diamond of forms cannot blow up.

Is it text at all? (and is it readable text?)

An empty TEXT pane is almost never a parser failure. It is a scan, or vector artwork, or fonts with no encoding. Guessing wrong at that moment is the difference between a tool that looks broken and a tool that explains your file to you — so both questions get answered from data the extractor already collected, with no second parse.

While interpreting each page, the interpreter keeps an operator census: text shows, font changes, images and their pixel area, inline images, path-painting operators, form runs. Classification is then arithmetic over counters:

The page-count threshold alone would call a one-line page “image-based”, so the classifier holds evidence a pure operator counter doesn’t: the characters that actually came out. A page that yielded nothing is never a text page, whatever its operator count.

The second question — is the text we got back really text? — is a character-class audit of the output, because mojibake looks exactly like success to everything downstream. Replacement characters in runs, private-use glyphs (codes with no meaning outside the font that drew them), C1 control codes, an implausibly low letter ratio, and the signature case: text sitting mostly in the high Latin-1 range with almost no ASCII letters, which is what glyph indices look like when decoded as if they were characters. Any of those raises a warning above the text instead of quietly handing you garbage.

Architecture: one pass, off the main thread, bounded

Three choices shape the module, and all three are about not making the browser do a second pass over a big file.

Everything runs in a Web Worker. The parse, the interpreter, the font decoding and the classification happen off the main thread, so a 400-page report never freezes the tab. The DOM-free modules (pdf-parse.js, pdf-text.js, pdf-fonts.js, pdf-quality.js) are also what the test suite imports directly in Node — the algorithms are testable without a browser.

The position model is built during the same pass that reads the text. There is no “extract, then lay out” second parse: every shown string becomes a positioned item as it is interpreted, and layoutText runs over that array. The same items are what multi-column reading order and a future Markdown converter need — the expensive part (running the content stream through the fonts) is already paid for.

Nothing is unbounded. A PDF is index-then-seek — its cross-reference table sits at the file’s tail — so a file up to 64 MB is read whole and anything larger is read through a bounded head (48 MB) + tail (4 MB) window, with objects seeked to by offset. On top of that the extractor stops at 500 pages or 2,000,000 characters, keeps at most 80,000 positioned items per page, and reports truncated: true rather than pretending it read everything. The quality audit samples 200,000 characters — enough to judge a book, cheap for a leaflet.

What still doesn’t work — honestly

• Multi-column reading order. Lines are grouped by baseline across the full page width, so a two-column layout interleaves: the sample paper’s left-hand citation sidebar merges into the abstract beside it, line by line. The position model is exactly what fixes this (cluster items into column bands, then read each band top-to-bottom) — it is the next step, not a solved one.
• Scans. A photographed page has no text in it at all. Nothing but OCR gets those words, and OCR is not what this tool does — it tells you how many pages would need it, and the IMAGES tab hands you the page scans to feed one.
• Text drawn as outlines. Some producers convert type to vector shapes, usually to dodge font licensing. That text was never text; the classifier names the case.
• Tables come out as rows of space-separated cells, which is often exactly what you want and sometimes not — see the figures, tables and metadata article.
• Right-to-left and vertical writing are laid out with the same left-to-right, top-to-bottom rules, which is wrong for Arabic, Hebrew and vertical CJK.

Saying which of these your file hit is, in practice, more useful than a tool that always produces something.

Sample paper: Dominianni C, Sinha R, Goedert JJ, Pei Z, Yang L, Hayes RB, Ahn J (2015), Sex, Body Mass Index, and Dietary Fiber Intake Influence the Human Gut Microbiome, PLoS ONE 10(4): e0124599 — an open-access article free of all copyright. “Attention Is All You Need” (Vaswani et al., 2017, arXiv:1706.03762) is not hosted here: that button downloads it from arxiv.org straight into your browser.