How to audit a 20 GB MDX file

MDX is three languages stacked in one file, and its compiler wants all of it in memory at once. But the question authors actually ask — which components does this use, and will they all resolve? — turns out not to need the document at all. This is a tour of the format, then the resumable scanner that answers it on a file far larger than RAM, with real JavaScript you can lift.

What is MDX?

MDX is Markdown that can import and render components. It was created by John Otander, Tim Neutkens, Guillermo Rauch and Brent Jackson, and released in 2018 as a way to stop documentation sites from choosing between “easy to write” and “can show a live demo”. MDX 2 (2022) rewrote the compiler on top of micromark and tightened the syntax considerably — most notably, JSX and expressions became first-class syntax rather than passed-through HTML.

If you have written docs for Docusaurus, Next.js, Astro, Storybook, Gatsby or Redwood, you have written MDX. And that is exactly how .mdx ends up in sizes nobody designed for: a docs site that concatenates every version of every page for a search index, an API reference generated from an OpenAPI spec with one section per endpoint, or a changelog assembled from ten years of releases.

Three languages in one file

Every byte of an MDX file belongs to exactly one of three layers, and knowing which one is the whole problem:

A small file with all four:

---
title: Shipping your first release
---

import Callout from '@site/src/components/Callout'
export const releasedOn = '2026-08-04'

# Shipping your first release

Last release: {releasedOn}.

<Callout type="warning">

Publishing is **not** reversible.

</Callout>

```js
// inside a fence, this is sample text — not an import
import NotReal from './nope'
```

Note the last block. That import is inside a fenced code block, so it is Markdown, not ESM. Any tool that greps for ^import gets this wrong, and so does any tool that treats <AlsoNotAComponent /> in a code sample as a component. Layer assignment is not optional.

The error everyone hits

By a wide margin the most common MDX build failure is this one:

Error: Expected component `Kbd` to be defined: you likely forgot to import,
pass, or provide it.

It is worth understanding why it is so easy to hit. MDX compiles your document into a JavaScript module. A capitalised tag like <Kbd> does not become an HTML element — it becomes a variable reference. Lowercase tags (<div>) become string literals passed to the JSX runtime; capitalised or dotted ones (<Kbd>, <Chart.Line>) become identifiers resolved from three places, in order:

1. an import in the file itself,
2. something the file exports (export const Kbd = …),
3. the components object supplied at render time by an MDXProvider or the components prop.

The catch is that none of that is checked when the file compiles. Compilation succeeds; the failure arrives when the module is rendered, which on a static site means somewhere in the middle of a build of two thousand pages. And because option 3 is invisible from inside the file, a linter that only looks at one document can never be sure — but it can tell you, with certainty, which tags rely on it. That list is the useful artefact, and it is what the COMPONENTS tab produces.

Why the compiler can't do this at scale

The obvious implementation is to run the real thing: @mdx-js/mdx parses to an mdast tree, transforms it to hast, then to estree, then prints JavaScript. Three whole-document trees, each with a node object per element, per text run, per attribute. The rule of thumb for that pipeline is 10–20× the source size in resident memory.

So a 50 MB MDX file wants a gigabyte. A 20 GB one wants a data centre. And you cannot fix it by chunking the compiler, because a tree parser is not resumable: hand it the second half of a document and it has no idea it is inside a <Tabs> element, inside a fenced code block, or halfway through a quoted attribute.

The component audit is a regular problem

Here is the observation the whole route is built on. Rendering MDX needs the tree. Auditing it does not.

To answer “which components are used, with which props, and which ones resolve” you need to know, at each character, only:

• whether you are inside a fenced code block (and which fence opened it),
• how deep the JSX element stack is, and what is on it,
• whether you are inside a tag, and if so inside a quoted attribute or a braced expression,
• whether you are inside a multi-line import,
• the set of bindings imported and exported so far, and the components seen so far.

The first four are a finite amount of state — a few hundred bytes, regardless of file size. The last two grow with the number of distinct components and imports, which in the largest real docs file is a few hundred entries. Nothing in that list is proportional to the document. So the audit can be a resumable state machine fed one line at a time, and its peak memory is one block of input plus a small table.

The resumable machine

The design is a two-level machine: a line classifier that decides which layer a line belongs to, wrapping a character walker that finds tags and expressions inside it. Both keep their state in the closure, so any of it can survive between calls.

Start with the buffering, because that is what makes push() accept an arbitrary slice:

function createMdxMachine(onEvent) {
  let buf = '';       // partial trailing line, not yet terminated
  let bufStart = 0;   // absolute offset of buf[0]
  let lineNo = 0;

  function push(text) {
    buf += text;
    let nl;
    while ((nl = buf.indexOf('\n')) !== -1) {
      const raw = buf.slice(0, nl);
      const start = bufStart;
      buf = buf.slice(nl + 1);
      bufStart = start + nl + 1;
      handleLine(raw.endsWith('\r') ? raw.slice(0, -1) : raw, start);
    }
  }

  function finish() {
    if (buf.length) handleLine(buf, bufStart);   // the last, unterminated line
    // … flush an open ESM statement, an unterminated frontmatter block …
  }

Everything downstream sees only whole lines, so no other piece of the machine has to think about block boundaries at all. That single decision is what makes the rest of it ordinary code.

Classifying a line

Order matters here, and each rule exists because of a specific way MDX bites:

function handleLine(text, start) {
  lineNo++;

  // 1. Frontmatter — only ever the very first line of the file.
  if (fmState === 0) {
    if (lineNo === 1 && /^?---\s*$/.test(text)) { fmState = 1; return; }
    fmState = 2;
  } else if (fmState === 1) { /* collect until --- or ... */ return; }

  // 2. A code fence wins over everything: ``` inside a <Tabs> block is
  //    still a code block, and <Foo/> inside one is sample code.
  if (fence) { /* close it if this line matches, emit, */ return; }
  if (!tag) { /* … open one if this line starts a fence … */ }

  // 3. ES modules. MDX requires them at column 0, and they never appear
  //    inside JSX children — so this test is exact, not a guess.
  if (esm) { /* continue a multi-line import until brackets balance */ return; }
  if (!tag && !expr && stack.length === 0 && /^(import|export)\b/.test(text)) {
    esm = { start, line: lineNo, lines: [text], depth: bracketDelta(text) };
    return;
  }

  // 4. Everything else: Markdown that may contain JSX and expressions.
  const plain = walk(text, start);
  emit({ t: 'line', text, plain, line: lineNo, start, jsxDepth: stack.length });
}

walk() returns the line with tags and expressions removed. That is what the Markdown counters and the heading extractor read, so ## Heading <Badge/> still counts as a heading and the badge does not leak into its text.

Walking the characters

The character walk is where the cross-line state earns its keep. A tag can be cut in half by a block boundary, and so can a quoted attribute inside it:

function walk(text, start) {
  let plain = '', i = 0;
  while (i < text.length) {
    if (tag)  { i = continueTag(text, i, start); continue; }   // carried in
    if (expr) { i = continueExpr(text, i, start); continue; }  // carried in
    const ch = text[i];

    // Inline code: `<Foo/>` in prose is a code span, not a component.
    if (ch === '`' && stack.length === 0) { /* skip the span */ continue; }

    // A tag: `<` immediately followed by a name, a `/`, or `>` (a fragment).
    if (ch === '<' && /[A-Za-z_$/>]/.test(text[i + 1] ?? '')) {
      startTag(start + i); i++;
      if (text[i] === '/') { tag.closing = true; i++; }
      continue;
    }
    if (ch === '{') { expr = { start: start + i, depth: 0, text: '' }; continue; }

    plain += ch; i++;
  }
  if (tag) tag.chars++;        // the newline is part of the tag
  if (expr) expr.text += '\n';
  return plain;
}

The stack.length === 0 guard on backticks is not decoration: inside a JSX element, a backtick is an ordinary character, and treating it as a code span there would swallow half a <Tabs> block.

Resolving a tag

Once a tag closes, resolution is a lookup, and it is the part with actual product value:

function resolveTag(tag, imports, localNames) {
  const root = tag.split('.')[0];                     // <Chart.Line> → Chart
  const imp = imports.get(root);
  if (imp) return { kind: 'import', from: imp.source };
  if (localNames.has(root)) return { kind: 'local' }; // export const Kbd = …
  if (!/^[A-Z]/.test(tag) && !tag.includes('.')) return { kind: 'html' };
  return { kind: 'undefined' };                       // ← the build error
}

The mirror image comes free: any import binding never marked used is an unused import. Marking is done from two places — a tag name, and every identifier inside an expression, since {formatDate(d)} uses formatDate without ever naming it as a tag.

Feeding it 20 GB

With a resumable machine, the reader is nine lines. Slice, decode, push, discard:

const AUDIT_BLOCK = 4 * 1024 * 1024;      // 4 MB

async function auditFile(file) {
  const scanner = createMdxScanner();
  const decoder = new TextDecoder('utf-8');
  let at = 0;

  while (at < file.size) {
    const end = Math.min(file.size, at + AUDIT_BLOCK);
    const bytes = new Uint8Array(await file.slice(at, end).arrayBuffer());
    // { stream: true }: a character split across this boundary is held back
    // and completed by the next block — no block is decoded in isolation.
    scanner.push(decoder.decode(bytes, { stream: true }));
    at = end;
    postMessage({ type: 'audit-progress', scanned: at, total: file.size });
  }
  scanner.push(decoder.decode());   // flush the decoder's held bytes
  scanner.finish();                 // flush the scanner's partial line
  return scanner.report();
}

file.slice() on a File does not read anything — it returns a lazy view, and the bytes are only pulled when arrayBuffer() is awaited. So at no point does more than one 4 MB block exist, and peak memory is that block plus the component table. The file can be 20 GB; it can be larger than the disk it is streamed from over the network.

Proving it works

A resumable scanner has one property worth testing above all others: the result must not depend on where the boundaries fell. That is a property test, not an example test, and it is short:

const oneShot = JSON.stringify(auditMdx(src));

for (let trial = 0; trial < 300; trial++) {
  const scanner = createMdxScanner();
  let i = 0;
  while (i < src.length) {
    const n = 1 + Math.floor(Math.random() * 60);   // random split points
    scanner.push(src.slice(i, i + n));
    i += n;
  }
  scanner.finish();
  assert.equal(JSON.stringify(scanner.report()), oneShot);
}

Chunks of one character are the interesting case: every tag, every attribute, every fence marker and every multi-byte character gets split. If the machine survives 300 random chunkings and a sweep of fixed sizes down to 1, the carried state is genuinely complete — and this test is what caught the fence marker and the quoted-attribute state during development.

What the streaming approach gives up

Streaming buys unbounded size by refusing to hold the document, and some things genuinely need it:

• Rendering. A preview is a tree by definition. OmniViewer's PREVIEW, FORMATTED and TOC tabs therefore work on a bounded prefix and say so on screen; only the audit is uncapped.
• The MDXProvider blind spot. A component supplied at render time is not visible from inside the file, so “unresolved” means “must come from a provider”, not “definitely broken”. That is a real distinction and the tab words it that way.
• Cross-file resolution. We can tell you a component is imported from @site/src/components/Callout; we cannot tell you that path exists. That needs the repository, not the file.
• Type-level truth. A prop list is what the document passes, gathered by observation. It is not the component's declared prop types.

That trade is the same split every format on OmniViewer uses: window what you view, bound what you parse — and stream the one question that can be answered without either.

OmniViewer opens every file format in your browser — Markdown, JSON, CSV, YAML, JavaScript and more — powered by the same windowed engine as fastjsonviewer.com and hugecsv.com. MDX is one of the formats with dedicated tooling.