Inside an MP4: boxes, moov vs mdat & why your video won't stream

An MP4 is not a stream of video frames — it's a tiny database bolted onto a huge blob. This is a tour of the ISO Base Media File Format: the box structure, the moov index that makes playback possible, the chunk-offset tables that make the file fragile, the metadata your camera hides in it — and the "fast start" byte surgery that makes a video stream, done in a way that works on a 20 GB file without ever reading the media.

Everything is a box

MP4 is the marketing name for the ISO Base Media File Format (ISO/IEC 14496-12), which is itself a light rebranding of Apple's QuickTime .mov container from 1991. QuickTime called the building block an atom; the ISO spec calls it a box. Same thing: a length-prefixed record.

┌────────────┬────────────┬─────────────────────────────┐
│ size (u32) │ type (4cc) │ payload (size − 8 bytes)    │
└────────────┴────────────┴─────────────────────────────┘

Three special cases keep parser authors honest: a size of 1 means "the real size is a 64-bit number in the next 8 bytes" (how a single mdat can exceed 4 GB); a size of 0 means "this box runs to the end of the file" (legal only for the last one); and some boxes are containers — their payload is more boxes. That recursion gives the file a tree:

ftyp                     file type & brands ("this is isom/mp42")
moov                     movie header — the index (small)
├── mvhd                 timescale & duration
├── trak                 one per track (video, audio, subtitles…)
│   ├── tkhd             track id, duration, pixel dimensions
│   └── mdia
│       ├── mdhd         media timescale, language
│       ├── hdlr         'vide' / 'soun' — what kind of track
│       └── minf → stbl  the sample table (see below)
└── udta → meta → ilst   iTunes-style metadata (title, artist, GPS…)
free                     padding
mdat                     the actual compressed media bytes (huge)

Reading one box header is ten lines of code, and because every box declares its size you can skip any box you don't care about — the property everything in this article hangs on:

function readBoxHeader(u8, off, end) {
  if (off + 8 > end) return null;
  let size = u32(u8, off);                    // big-endian
  const type = ascii(u8, off + 4, 4);         // 'moov', 'mdat', …
  let headerSize = 8;
  if (size === 1) { size = u64(u8, off + 8); headerSize = 16; }
  else if (size === 0) { size = end - off; }  // "to end of file"
  if (size < headerSize || off + size > end) return null; // corrupt
  return { type, size, headerSize };
}

moov: a database about mdat

The genius (and the curse) of the format is the split between mdat and moov. mdat is nothing but concatenated compressed frames — no sync markers, no headers, no way to find frame 500 by looking at it. All of the structure lives in moov's sample tables (stbl), one set per track:

To show frame 500, a player runs the lookup chain: stts (which sample is at t=16.7 s) → stsc (which chunk holds sample 500) → stco (that chunk starts at byte 48,213,992) → stsz (skip 3 samples of known size, read 41,207 bytes) → hand them to the decoder. The media blob is only ever accessed through this index — which is why moov stays small (an index over frames, kilobytes to a few megabytes) no matter how big the movie is, and why a file with a broken moov is dead even though 99.9% of its bytes are intact.

stco/co64: absolute offsets, the fragile part

Here's the design decision that makes MP4s brittle: stco chunk offsets are absolute file positions, not offsets relative to mdat. Insert, remove or resize any byte before mdat and every entry in every track's stco is now wrong — the file plays garbage or nothing.

stco payload:                          co64 payload:
version/flags  u32                     version/flags  u32
entry_count    u32                     entry_count    u32
offset[0]      u32  ← absolute!        offset[0]      u64
offset[1]      u32                     offset[1]      u64
…                                      …

stco entries are 32-bit, so they can't point past 4 GB; files bigger than that need the 64-bit twin co64. That distinction matters the moment you start moving boxes around — as we're about to.

Metadata — and the GPS atom you forgot about

Tags live at moov/udta/meta/ilst, in iTunes-style items: ©nam (title), ©ART (artist), ©alb (album), ©day (date), ©too (encoder), covr (cover art) — each wrapping a data box with a type indicator (1 = UTF-8 text) and the payload. Two boxes deserve special attention:

• ©xyz — the GPS position where the video was recorded, as an ISO 6709 string like +48.8577+002.2950/. Phones write it by default; most people have no idea it's in every clip they share.
• meta itself is a trap: the ISO spec makes it a FullBox (4 version/flags bytes before its children) but classic QuickTime files omit those bytes. Parsers disambiguate by checking whether hdlr appears 8 or 12 bytes in.

Because ilst lives inside moov, editing a tag resizes moov — which, per the previous section, shifts mdat and invalidates every chunk offset. Even "just change the title" is offset-patching surgery. The /mp4 toolkit's METADATA tab does exactly that (and offers one-click removal of ©xyz); the media bytes are never touched, never re-encoded.

Fast start: why your video won't stream

An encoder can't know the sample tables until it has encoded the last frame, so the lazy (and default, for many tools) layout writes moov last:

ftyp · mdat (20 GB) · moov        ← progressive playback impossible
ftyp · moov · mdat (20 GB)        ← plays after ~one round trip

A browser handed the first layout must download all 20 GB before it can even learn what codec the file uses. That's the whole "fast start" problem: the index has to arrive before the data it indexes. It's why ffmpeg has -movflags +faststart and why qt-faststart exists as a standalone tool. The catch: moving moov to the front shifts mdat by exactly moov's size, so every stco/co64 entry must be patched.

The byte surgery

The fix never touches the media. Re-serialize moov (with any tag edits), decide the new box order, compute how far every top-level box moved, then add each box's shift to every chunk offset that points into it:

// layout: the top-level boxes in file order [{type, start, size}]
// 1. new order: ftyp · moov · everything else, original order kept
const others = layout.filter((b) => b.type !== 'moov');
const ordered = [others[0] /* ftyp */, moov, ...others.slice(1)];

// 2. how far did each box move?
let pos = 0;
const shifts = new Map();
for (const b of ordered) { shifts.set(b.start, pos - b.start); pos += b.size; }

// 3. patch every chunk offset by the shift of the box it points into
const shiftFor = (v) => {
  for (const b of others) if (v >= b.start && v < b.start + b.size)
    return shifts.get(b.start);
  return 0; // dangling offset in an already-broken file: leave it
};
for (const table of chunkTables)           // every stco/co64, every track
  for (let i = 0; i < table.count; i++)
    table.set(i, table.get(i) + shiftFor(table.get(i)));

One subtlety: pushing a multi-gigabyte mdat a few kilobytes further down the file can carry a 32-bit stco entry past the 4 GB line. The rewrite must then upgrade that table to co64 — which widens every entry from 4 to 8 bytes, which grows moov, which shifts mdat a little more… so the algorithm iterates until the upgrade set is stable (it converges in a pass or two).

Scaling to 20 GB: never read what you don't parse

Everything above needs surprisingly few bytes. The top-level walk reads 16 bytes per box and skips the rest — for a 20 GB movie that's four or five tiny reads, because mdat is skipped in one hop using its declared size:

async function topLevelBoxes(file) {         // file: a Blob/File handle
  const layout = [];
  let off = 0;
  while (off + 8 <= file.size) {
    const head = new Uint8Array(await file.slice(off, off + 16).arrayBuffer());
    const h = decodeHeader(head, file.size - off);   // 32/64-bit/to-EOF sizes
    if (!h) break;                                   // malformed: stop cleanly
    layout.push({ type: h.type, start: off, size: h.size });
    off += h.size;                                   // ← skips 20 GB of mdat
  }
  return layout;
}

Then moov — remember, an index, not media — is read in full (megabytes at worst) and parsed in memory, off the main thread. The rewrite output is assembled the same way: the new moov bytes plus slices of the original file, so the browser streams the download and the 20 GB of media is never in RAM:

// parts: [{bytes}, {start, end}, …] — fresh moov + original slices
const blob = new Blob(parts.map((p) =>
  p.bytes ? p.bytes : file.slice(p.start, p.end)), { type: 'video/mp4' });
// Blob slices are lazy references, not copies: downloading this Blob
// streams straight from disk to disk.

This is the same discipline OmniViewer applies to 20 GB YAML and multi-gigabyte MP3s: read the bytes that carry structure, slice the bytes that don't.

Fragmented MP4: the other layout

Streaming platforms sidestep the fast-start problem entirely with fragmented MP4 (fMP4/CMAF): a skeleton moov up front, then repeating moof+mdat pairs, each fragment carrying its own mini-index. That's what DASH and HLS actually deliver, and it's why you can't run relocation surgery on one — moof and sidx boxes carry their own offset bookkeeping that a box shuffle would break. A well-behaved tool detects moof/sidx and refuses the rewrite instead of corrupting the file; it's already streamable by construction.

See it on a real file