How to read a 20 GB XML file

A 20 GB XML file will never fit in memory, and the usual way to read XML — build a DOM tree — needs several times the file size in RAM. This is a tour of the format and how parsers read it, the split between DOM and SAX, and the streaming, memory-bounded algorithm that reads a file far larger than RAM, with real JavaScript you can lift.

On this page What is XML The grammar DOM vs SAX Why 20 GB is hard Reading only what you paint A streaming parser The full algorithm Caveats

What is XML?

XML — the “Extensible Markup Language” — is a text format for storing and transporting structured data using tags you define yourself. It was standardised by the W3C in 1998 as a simplified descendant of SGML, meant to be strict enough for machines yet still readable by people. Where HTML has a fixed vocabulary of tags aimed at display, XML lets you invent tags that describe data.

For a decade XML was the backbone of the web's data plumbing, and it is still everywhere: RSS and Atom feeds, SVG graphics, office documents (.docx, .xlsx are zipped XML), SOAP web services, Android layouts, Maven pom.xml files and endless configuration. And that is exactly how files get enormous: a database exported one <record> per row, an aggregated log turned structured, or a full data feed can run to many gigabytes in a single document.

The grammar, in one screen

An XML document is a tree with exactly one root element, optionally preceded by a prolog (the XML declaration, comments, a doctype). Every node is one of a small set of kinds:

Node kind	Looks like
Element	`<title>…</title>` or self-closing `<br/>`
Attribute	`id="7"` inside an element's start tag
Text	character data between tags
CDATA	`<![CDATA[ raw & unescaped ]]>`
Comment / PI	`<!-- … -->` · `<?target …?>`

Structure comes from nesting: every start tag must have a matching end tag, and tags must nest cleanly — <a><b></a></b> is forbidden. A document that obeys these rules is well-formed. A few more essentials:

<?xml version="1.0" encoding="UTF-8"?>   <!-- the prolog -->
<catalog xmlns:x="http://example.com/ns">  <!-- namespace on the root -->
  <book id="bk101" lang="en">            <!-- attributes -->
    <title>Streaming XML</title>         <!-- text content -->
    <price currency="USD">39.95</price>
    <x:note><![CDATA[ raw < & > kept verbatim ]]></x:note>
    <tags><tag>xml</tag><tag>parsing</tag></tags>
    <cover/>                             <!-- empty / self-closing -->
  </book>
</catalog>

Five entities are predefined — <, >, &, ", ' — plus numeric character references like © or ©. A namespace (xmlns) qualifies names with a prefix so vocabularies can be mixed without collisions, which is why feed formats stack itunes:, atom: and content: tags in one file.

DOM vs SAX — two ways to read a tree

There have always been two schools of XML parsing, and the difference is the whole story for large files:

DOM reads the entire document and builds a tree of node objects in memory. You get random access — walk to any element, query with XPath, edit and re-serialise. The cost: the tree is several times the file's byte size, so DOM is a non-starter past a few hundred megabytes.
SAX (and the closely related pull parser) never builds a tree. It scans the bytes once and emits a stream of tiny events — startElement, attribute, text, endElement — that your code handles and forgets. Memory stays flat no matter how big the file is; the price is that you only ever see a sliding window, never the whole tree at once.

For 20 GB, SAX-style streaming is the only option. The rest of this page builds one.

Why 20 GB is the hard part

The obvious approach — read the whole file → build the DOM → walk the tree — dies twice over at 20 GB:

The bytes don't fit. 20 GB of markup won't sit in a browser tab's memory, and the decoded UTF-16 string a language hands you is often two to four times the byte size again.
The DOM doesn't fit either. Even if the text fit, materialising every element, attribute and text node as live objects — each with parent, child and sibling links — multiplies the footprint several times over. A 20 GB document can be hundreds of millions of nodes.

You can resume XML from a tag boundary more easily than an indentation-based format, but a raw byte offset still doesn't tell you which elements are open around you, so naive random access into the middle doesn't give you a valid context. The way out is to never hold the whole thing at once. Two techniques do that, and OmniViewer uses both.

Technique 1 — read only the bytes you paint

To simply look at a huge file, you don't need to parse it at all — you need the handful of kilobytes currently on screen. A File from a drop or file picker is a handle to bytes on disk, and file.slice(start, end) returns a view without reading anything. You read a slice only when you scroll to it:

// A windowed reader: fetch just the byte range a viewport needs.
async function readWindow(file, startByte, endByte) {
  const blob = file.slice(startByte, endByte);          // no I/O yet
  const buf  = await blob.arrayBuffer();                // reads ~the window only
  return new TextDecoder('utf-8').decode(new Uint8Array(buf));
}

// Painting line N means: know where line N starts, read a screen's worth.
const screen = await readWindow(file, lineOffset[N], lineOffset[N] + 64 * 1024);

This is how OmniViewer's raw and hex views open a 20 GB XML as fast as a 2 KB one: memory stays proportional to the window, not the file. (A background pass builds a sparse index of line offsets so scrolling is instant, but it stores offsets, not content.)

Technique 2 — a streaming, event-emitting parser

Viewing is one thing; parsing 20 GB into structure without holding it all is the real prize — that is SAX, and it is simpler to build than it sounds. XML has a lovely property for streaming: the top-level syntax is a flat alternation of text runs and markup (anything from < to the matching >). A tokenizer that walks the bytes and, at each <, decides “start tag / end tag / comment / CDATA / PI” turns an unbounded byte stream into a bounded, one-token-at-a-time problem.

Two ideas make it work:

Tokenize, don't tree-build. Scan for the next <, emit the text before it, classify the markup, emit an event, repeat. Nothing accumulates unless you choose to keep it.
Track depth with a name stack. Push an element name on a start tag, pop it on the matching end tag. The stack is only ever as deep as the document nests — a few dozen frames — so memory is O(depth), not O(file). It also lets you detect mismatched nesting as you go.

The tokenizer

Feed it text blocks; it emits text and markup tokens, holding a small carry buffer for a token that straddles a block boundary:

// Feed it file blocks; it calls emit(kind, value) for each complete token.
function createXmlTokenizer(emit) {
  let carry = '';
  const flush = (buf, final) => {
    let i = 0;
    while (i < buf.length) {
      if (buf[i] === '<') {
        const end = buf.indexOf('>', i);
        if (end === -1) break;               // markup spills into the next block
        emit('markup', buf.slice(i, end + 1));
        i = end + 1;
      } else {
        let lt = buf.indexOf('<', i);
        if (lt === -1) { if (!final) break; lt = buf.length; }
        const text = buf.slice(i, lt);
        if (text.trim()) emit('text', text);
        i = lt;
      }
    }
    return buf.slice(i);                      // unconsumed tail waits for more bytes
  };
  return {
    push(chunk) { carry = flush(carry + chunk, false); },
    end() { flush(carry, true); carry = ''; },
  };
}

The event parser

Now classify each markup token and reconcile it against a stack of open elements. Start tags push and emit; end tags pop and emit; everything else is a leaf event. Every branch touches only the stack:

function createXmlEventParser(emit) {
  const stack = [];                                  // open element names

  return function token(kind, value) {
    if (kind === 'text') { emit('text', decodeEntities(value)); return; }

    if (value.startsWith('<!--') || value.startsWith('<?') ||
        value.startsWith('<!DOCTYPE')) return;          // comment / PI / doctype: skip
    if (value.startsWith('<![CDATA[')) {
      emit('text', value.slice(9, -3)); return;        // raw, unescaped
    }
    if (value.startsWith('</')) {                       // end tag
      const name = value.slice(2, -1).trim();
      while (stack.length && stack.pop() !== name) {}    // recover from bad nesting
      emit('endElement', name); return;
    }
    // start tag (maybe self-closing)
    const selfClose = value.endsWith('/>');
    const inner = value.slice(1, selfClose ? -2 : -1).trim();
    const name  = inner.split(/\s/, 1)[0];
    emit('startElement', name, parseAttrs(inner.slice(name.length)));
    if (selfClose) emit('endElement', name);
    else stack.push(name);
  };
}

Because the parser never keeps more than the current path from root to cursor, its memory is O(nesting depth) — a few dozen frames — no matter whether the file is 2 KB or 20 GB. The events stream past; you decide what, if anything, to retain.

Wiring it together

Block reader → tokenizer → event parser, driven by file.slice so the browser only ever holds one block plus the stack:

async function streamXml(file, emit, blockSize = 256 * 1024) {
  const parser    = createXmlEventParser(emit);
  const tokenizer = createXmlTokenizer(parser);
  const decoder   = new TextDecoder('utf-8');         // handles multi-byte splits

  for (let pos = 0; pos < file.size; pos += blockSize) {
    const buf = await file.slice(pos, pos + blockSize).arrayBuffer();
    tokenizer.push(decoder.decode(buf, { stream: true }));  // keep partial UTF-8
  }
  tokenizer.end();
}

// Example: count <item> elements in a 20 GB feed without building anything.
let items = 0;
await streamXml(file, (event, name) => {
  if (event === 'startElement' && name === 'item') items++;
});
console.log(items, 'items — peak memory: one 256 KB block + the stack');

That loop reads the file in 256 KB blocks (the same size OmniViewer streams for every format), so peak memory is a quarter-megabyte plus a shallow stack — whether the file is a kilobyte or twenty gigabytes. Want a filtered subset instead of a count? Keep only the elements matching your path and drop the rest; the file size stops mattering.

Note the multi-byte care. A 256 KB block can end in the middle of a UTF-8 character. TextDecoder with { stream: true } holds the incomplete tail bytes until the next block completes them — skip that and you'll corrupt every character that straddles a block boundary.

What the streaming approach gives up

Streaming buys unbounded size by refusing to hold the whole document, and a few things assume you can hold it:

XPath over the whole tree. An expression like //book[last()] or one that walks back up with .. needs a tree to query. Over a stream you can match forward, path-based patterns as events fly past, but not arbitrary backward or whole-document queries without retaining nodes.
Whole-document validation. You learn a document is malformed when you reach the bad tag, not up front — fine for streaming, but you can't answer “is this entire file well-formed?” without reading all of it.
Random access. To jump to record 9,000,000 you must either stream past the first 8,999,999 or have pre-built an index of their byte offsets.

That trade is why OmniViewer's XML tab parses a generous bounded prefix in memory (so the formatter, JSON conversion, tree, XPath and stats are instant and complete for normal files, and it tells you when a giant file was capped), while the raw and hex views stay fully windowed and open any size. It's the same split every format on the site uses: window what you view, bound what you parse.

Open the XML viewer →Pretty-print, convert to JSON, browse the tree, run XPath and read stats — locally, no upload. How we auto-detect formats →The content-sniffing algorithm that recognizes pasted or typed XML, JSON, CSV and more.

OmniViewer opens every file format in your browser — JSON, CSV, YAML, Markdown, HTML, JavaScript and more — powered by the same windowed engine as fastjsonviewer.com and hugecsv.com. XML is one of the formats with dedicated tooling.