How to read a 20 GB YAML file

A 20 GB YAML file will never fit in memory, and YAML is indentation-significant — you can't just seek to the middle and start reading. This is a tour of the format and its grammar, then the streaming, memory-bounded algorithm that reads a file far larger than RAM, with real JavaScript you can lift.

What is YAML?

YAML — a recursive acronym for “YAML Ain't Markup Language” — is a human-friendly data-serialization format. It was designed by Clark Evans, Ingy döt Net and Oren Ben-Kiki and first announced in 2001. Where JSON leans on brackets and quotes, YAML leans on indentation and line structure, which is what makes it pleasant to hand-write and hand-read.

The current specification is YAML 1.2 (2009). Its most important property for us: YAML 1.2 is a strict superset of JSON. Every valid JSON document is also valid YAML, so a YAML parser is, among other things, a JSON parser that also understands the indentation-based sugar.

YAML became the default language of configuration. If you've written a Kubernetes manifest, a GitHub Actions or GitLab CI pipeline, an Ansible playbook or a Docker Compose file, you've written YAML. And that's exactly how files get enormous: exported cluster state, aggregated CI logs turned structured, or a database dumped as one document per record can run to many gigabytes.

The grammar, in one screen

A YAML stream is one or more documents, separated by --- and optionally ended by .... Each document is a single node, and a node is one of three things:

Structure comes from indentation: a node nested under a key or a - is written more indented than its parent. Tabs are forbidden for indentation — spaces only. A few more essentials:

mapping:              # a comment runs to end of line
  key: value          # nesting by indentation
  list:
    - one             # a sequence, one item per '- '
    - two
  nested: {a: 1, b: 2}         # flow mapping
  inline: [red, green, blue]   # flow sequence

quoted:  "with \n escapes"     # double: C-style escapes
literal: 'it''s raw'           # single: '' is a literal quote

block_literal: |      # '|' keeps newlines verbatim
  line one
  line two
block_folded: >       # '>' folds newlines into spaces
  this becomes
  one long line

anchored: &id { host: a, port: 80 }   # define an anchor
alias:    *id                          # reuse it (a reference)
merged:
  <<: *id           # merge key: fold the mapping in
  port: 443       # local keys win

typed: [null, ~, true, false, 0x1F, 3.14, .inf]  # core-schema scalars

Scalar typing follows the core schema: null/~ is null; true/false (and yes/no, on/off in YAML 1.1) are booleans; things that look like integers or floats become numbers (including 0x hex, .inf, .nan); everything else is a string. Explicit tags like !!str 12 can force a type.

Why 20 GB is the hard part

The obvious approach — read the whole file → parse into objects → walk the objects — dies twice over at 20 GB:

• The bytes don't fit. 20 GB of text won't sit in a browser tab's memory, and the decoded UTF-16 string a language hands you is often two to four times the byte size again.
• The parse tree doesn't fit either. Even if the text fit, materializing every mapping, array and string as live objects multiplies the footprint. A 20 GB document can be hundreds of millions of nodes.

And unlike a line-oriented format such as CSV or NDJSON, you can't seek into the middle of YAML and start parsing. Meaning depends on indentation relative to ancestors that may be gigabytes earlier, so a byte offset alone doesn't tell you how deep you are. That rules out naive random access.

The way out is to never hold the whole thing at once. Two techniques do that, and OmniViewer uses both.

Technique 1 — read only the bytes you paint

To simply look at a huge file, you don't need to parse it at all — you need the handful of kilobytes currently on screen. A File from a drop or file picker is a handle to bytes on disk, and file.slice(start, end) returns a view without reading anything. You read a slice only when you scroll to it:

// A windowed reader: fetch just the byte range a viewport needs.
async function readWindow(file, startByte, endByte) {
  const blob = file.slice(startByte, endByte);          // no I/O yet
  const buf  = await blob.arrayBuffer();                // reads ~the window only
  return new TextDecoder('utf-8').decode(new Uint8Array(buf));
}

// Painting line N means: know where line N starts, read a screen's worth.
const screen = await readWindow(file, lineOffset[N], lineOffset[N] + 64 * 1024);

This is how OmniViewer's raw and hex views open a 20 GB YAML as fast as a 2 KB one: memory stays proportional to the window, not the file. (A background pass builds a sparse index of line offsets so scrolling is instant, but it stores offsets, not content.)

Technique 2 — a streaming, event-emitting parser

Viewing is one thing; parsing 20 GB into structure without holding it all is the real prize. The trick is to parse as a stream of events rather than build one giant tree — the SAX approach, applied to YAML. You feed the parser bytes in fixed-size blocks and it emits tiny events — mappingStart, key, scalar, sequenceStart, end — that your code handles and forgets. Nothing accumulates unless you choose to keep it.

Two ideas make this work for an indentation-based grammar:

1. Split the stream into lines, not objects. YAML's block structure is line-oriented, so a lexer that yields one logical line at a time (carrying its indentation and content) turns an unbounded byte stream into a bounded, one-line-at-a-time problem.
2. Track depth with an indentation stack. Each line's leading-space count, compared with the stack of open containers, tells you whether you're going deeper (push), staying level (a sibling), or popping out (close one or more containers). That's the whole secret to parsing indentation with O(depth) memory — not O(file).

The line lexer

First, turn incoming blocks into complete lines. Hold a small carry buffer for the partial last line of each block; emit only whole lines:

// Feed it file blocks; it calls onLine(text) for each complete line.
function createLineLexer(onLine) {
  let carry = '';
  return {
    push(chunkText) {
      const text = carry + chunkText;
      let start = 0, nl;
      while ((nl = text.indexOf('\n', start)) !== -1) {
        onLine(text.slice(start, nl).replace(/\r$/, ''));
        start = nl + 1;
      }
      carry = text.slice(start);            // partial line waits for more bytes
    },
    end() { if (carry) onLine(carry.replace(/\r$/, '')); carry = ''; },
  };
}

The indentation stack

Now the core. For each line, measure its indent, strip a trailing comment, then reconcile it against a stack of open containers. Deeper → open a child; shallower → close containers until the indents line up; same indent → a sibling. Every branch emits an event and touches only the stack:

function createYamlEventParser(emit) {
  const stack = [];                    // open containers: { indent, kind }

  const closeTo = (indent) => {
    while (stack.length && stack[stack.length - 1].indent >= indent) {
      const top = stack.pop();
      emit(top.kind === 'seq' ? 'sequenceEnd' : 'mappingEnd');
    }
  };

  return function line(raw) {
    if (/^\s*$/.test(raw) || /^\s*#/.test(raw)) return;   // blank / comment
    if (/^---/.test(raw)) { closeTo(0); emit('documentStart'); return; }

    const indent = raw.length - raw.replace(/^ +/, '').length;
    const text   = stripComment(raw.trim());
    closeTo(indent + 1);                 // pop anything not an ancestor

    if (text.startsWith('- ')) {         // sequence item
      if (top()?.kind !== 'seq' || top().indent < indent) {
        stack.push({ indent, kind: 'seq' }); emit('sequenceStart');
      }
      emitValue(text.slice(2), indent + 2);
    } else {                             // mapping "key: value"
      const c = splitKeyValue(text);
      if (top()?.kind !== 'map' || top().indent < indent) {
        stack.push({ indent, kind: 'map' }); emit('mappingStart');
      }
      emit('key', c.key);
      if (c.value !== '') emitValue(c.value, indent);   // else child block follows
    }
  };

  function top() { return stack[stack.length - 1]; }
  function emitValue(v, indent) {
    if (v === '|' || v === '>') return;             // block scalar: gather next lines
    emit('scalar', resolveScalar(v));               // type it (null/bool/number/string)
  }
}

Because the parser never keeps more than the current path from root to cursor, its memory is O(nesting depth) — a few dozen frames — no matter whether the file is 2 KB or 20 GB. The events stream past; you decide what, if anything, to retain.

Wiring it together

Block reader → line lexer → event parser, driven by file.slice so the browser only ever holds one block plus the stack:

async function streamYaml(file, emit, blockSize = 256 * 1024) {
  const parser  = createYamlEventParser(emit);
  const lexer   = createLineLexer(parser);
  const decoder = new TextDecoder('utf-8');           // handles multi-byte splits

  for (let pos = 0; pos < file.size; pos += blockSize) {
    const buf = await file.slice(pos, pos + blockSize).arrayBuffer();
    lexer.push(decoder.decode(buf, { stream: true }));  // {stream:true} = keep partial UTF-8
  }
  lexer.end();
}

// Example: count keys in a 20 GB file without building anything.
let keys = 0;
await streamYaml(file, (event) => { if (event === 'key') keys++; });
console.log(keys, 'keys — peak memory: one 256 KB block + the stack');

That loop reads the file in 256 KB blocks (the same size OmniViewer streams for every format), so peak memory is a quarter-megabyte plus a shallow stack — whether the file is a kilobyte or twenty gigabytes. Want a filtered subset instead of a count? Keep only the events matching your path and drop the rest; the file size stops mattering.

What the streaming approach gives up

Streaming buys unbounded size by refusing to hold the whole document, and a few YAML features assume you can look back:

• Aliases across huge distances. An *alias refers to an earlier &anchor. If the anchored value was gigabytes ago and you didn't retain it, you can only emit a reference, not re-inline the value. In practice anchors are small and local, so keeping a bounded map of them is cheap.
• Whole-document validation. You learn a document is malformed when you reach the bad line, not up front — fine for streaming, but you can't answer “is this entire file valid?” without reading all of it.
• Random access. To jump to record 9,000,000 you must either stream past the first 8,999,999 or have pre-built an index of their offsets.

That trade is why OmniViewer's YAML tab parses a generous bounded prefix in memory (so the JSON conversion, tree and stats are instant and complete for normal files, and it tells you when a giant file was capped), while the raw and hex views stay fully windowed and open any size. It's the same split every format on the site uses: window what you view, bound what you parse.

OmniViewer opens every file format in your browser — JSON, CSV, Markdown, HTML, JavaScript and more — powered by the same windowed engine as fastjsonviewer.com and hugecsv.com. YAML is one of the formats with dedicated tooling.