Inside a HEIC file: boxes, items & HEVC

A .heic is what your phone saves photos as, and it is quietly one of the more sophisticated image formats in wide use: the pixels are coded with HEVC (H.265), and the file that wraps them is not an image container at all but a general-purpose ISO-BMFF box tree — the very same grammar as MP4. Where MP4 describes a timeline of tracks, HEIF describes a set of items. This is a tour of that structure: the ftyp brand that names it, the meta box that lists every item, where the dimensions and the HEVC profile live, where EXIF and the GPS position hide — and the JavaScript to strip that metadata without touching a coded pixel and to map a multi-gigabyte HEIC from just its front matter.

Everything is a box

HEIF inherits the ISO Base Media File Format (ISO/IEC 14496-12), the container designed for MP4. A file is a flat sequence of boxes (QuickTime calls them atoms). Each box is a 4-byte big-endian size, a 4-byte ASCII type, then the payload — and for a container box the payload is itself more boxes:

00 00 00 18  66 74 79 70  ...        ← box: size 0x18 = 24 bytes, type "ftyp"
             └─ type        └─ 16 bytes of payload

size == 1  → a 64-bit size follows the type (an 8-byte "largesize")
size == 0  → the box runs to end-of-file (only valid for the last box)

Reading a header is a dozen lines, and it is identical to the one an MP4 parser uses:

function readBox(u8, off, end) {
  if (off + 8 > end) return null;
  let size = u32(u8, off);
  const type = str(u8, off + 4, 4);
  let headerSize = 8;
  if (size === 1) { size = u64(u8, off + 8); headerSize = 16; } // 64-bit size
  else if (size === 0) size = end - off;                        // to EOF
  return { type, size, headerSize };
}

The top level of a HEIC is short and predictable: an ftyp, then a meta box holding the whole structure, then an mdat holding the coded bytes. The mdat is the bulk of the file and — exactly like MP4’s mdat or PNG’s IDAT — you never need to read it to understand the image.

The ftyp brand tells you it's a HEIC

Because HEIC and MP4 share the ISO-BMFF grammar, they share a magic number: both begin with an ftyp box at offset 4. The brand inside it disambiguates them. The first four bytes of the ftyp payload are the major brand, followed by a minor version and a list of compatible brands:

const p = ftyp.offset + ftyp.headerSize;
const major = str(u8, p, 4);            // "heic", "mif1", "msf1", "avif"…
const compatible = [];
for (let o = p + 8; o + 4 <= ftyp.offset + ftyp.size; o += 4) compatible.push(str(u8, o, 4));

The HEVC-image brands are heic/heix (single image), the hev* set (image sequences), and the generic HEIF brands mif1/msf1. An avif major brand is the same container with AV1-coded images instead of HEVC. Detecting the family from the brand — never the file extension — is how OmniViewer routes a .heic renamed .mp4 to the right toolkit anyway.

The meta box: a picture as a set of items

Here is where HEIF diverges from MP4. Instead of tracks, the meta box (a FullBox, so its payload starts with a 4-byte version/flags) describes the file as a set of items. Its children are the pieces of a small database:

Parsing an infe (version 2 is what phones write) gives you the item’s identity:

// infe payload after version/flags: item_ID(16), protection(16), item_type(4), name\0
const id   = u16(u8, p);
const type = str(u8, p + 4, 4);           // "hvc1" | "Exif" | "grid" | "mime"…
const name = cstr(u8, p + 8);             // null-terminated UTF-8

Join those with the references and you can label every item by role. A reference points from a subordinate item to its master — a thmb goes from the thumbnail to the full image, a cdsc goes from the Exif to the image it describes — which is the detail most first-time parsers get backwards. A typical iPhone photo yields something like:

#1  grid   primary image · tiled grid   4032×3024
#2  hvc1   grid tile                    2016×1512
#3  hvc1   grid tile                    2016×1512
#4  hvc1   grid tile                    2016×1512
#5  hvc1   grid tile                    2016×1512
#6  hvc1   thumbnail                     320×240
#7  Exif   EXIF metadata                 —          ← camera fields + GPS

Dimensions and the HEVC profile, without decoding

You never decode HEVC to learn the image’s shape — that lives in properties. Two matter most. ispe (image spatial extents) is the pixel size:

// ispe is a FullBox: skip 4 bytes of version/flags, then two 32-bit dimensions
const width  = u32(u8, p + 4);
const height = u32(u8, p + 8);

And hvcC (the HEVC configuration) carries the profile, tier and level in a fixed layout — byte 1 packs the profile, byte 12 is the level:

const profileIdc = u8[p + 1] & 0x1f;      // 1 = Main, 2 = Main 10, 3 = Main Still…
const tier       = (u8[p + 1] >> 5) & 1;  // 0 = Main, 1 = High
const level      = u8[p + 12] / 30;        // 93 → level 3.1

A third, irot, is a single byte whose low two bits are the counter-clockwise rotation in 90° steps — the reason a photo can look sideways in a viewer that ignores it. The ipma box ties these to items: for each item it lists the 1-based indices of the ipco children that apply, so item 1’s [1,2,3] means “the first ispe, the pixi, the hvcC.”

A grid primary item usually has no ispe of its own; its output size lives in the item’s own tiny payload — rows−1, cols−1, then the full width and height — which is why most iPhone HEICs are a mosaic of four or more HEVC tiles stitched into one picture.

Where an item's bytes live

The iloc box is the fiddliest but the most important: it is the map from an item ID to the actual bytes. Its header declares how many bytes each field takes (a nibble each for offset, length and base-offset sizes), then for every item a base offset and a list of extents (offset + length pairs) into the file. A construction_method of 0 means the offsets are file-relative; 1 means they are relative to an inline idat box inside meta. Resolve it and you have an item’s exact byte range:

function itemRange(loc) {
  const anchor = loc.constructionMethod === 0
    ? loc.baseOffset                       // file-relative
    : idat.offset + loc.baseOffset;        // idat-relative
  return loc.extents.map(e => ({ start: anchor + e.offset, length: e.length }));
}

Where EXIF and GPS live

In a JPEG the EXIF is a marker segment; in a HEIC it is a first-class item — an infe of type Exif, referenced by a cdsc from the item it describes, and located by iloc. Its payload is a 4-byte offset to the TIFF header followed by exactly the same TIFF/EXIF block a JPEG carries, so the same IFD reader decodes it:

const [range] = itemRange(iloc.get(exifItemId));
const bytes   = file.slice(range.start, range.start + range.length); // a small, targeted read
const tiff    = bytes.subarray(4 + u32(bytes, 0));   // skip to the "II"/"MM" TIFF header
const exif    = parseExif(tiff);   // make, model, dates… and the GPS lat/long

That GPS pair — latitude and longitude as three RATIONALs each, degrees/minutes/seconds — is the single most sensitive thing a photo carries: it is where you were standing. Most people have no idea their phone embeds it.

Stripping metadata in place

Removing an item cleanly from an ISO-BMFF file is deceptively hard: iloc offsets are absolute file positions, so deleting the Exif item’s reference and its bytes would shift everything after it and invalidate every other offset. The robust, streamable answer is not to move anything at all — overwrite the metadata item’s bytes with zeros where they sit. Every box offset and every iloc entry stays exactly valid; the GPS coordinates are physically gone; and the coded image is copied through byte-for-byte:

// The regions to zero: each metadata item's extents, as absolute ranges.
const holes = [];
for (const it of items)
  if (it.type === 'Exif' || isXmp(it)) holes.push(...itemRange(iloc.get(it.id)));

// Emit copy-slice / zero-fill parts that exactly tile [0, fileSize) so the
// output is byte-identical except inside those holes.
const parts = [];
let cursor = 0;
for (const h of holes.sort((a, b) => a.start - b.start)) {
  if (h.start > cursor) parts.push(file.slice(cursor, h.start));  // untouched
  parts.push(new Uint8Array(h.length));                          // zeros
  cursor = h.start + h.length;
}
parts.push(file.slice(cursor));               // the rest, incl. the whole mdat
new Blob(parts, { type: 'image/heic' });      // streams — nothing large in memory

Because the parts are mostly Blob slices of the original File, the browser streams them straight to disk; a 40 MB burst frame or a 2 GB Live Photo is cleaned without ever loading its pixels into memory.

Why the preview is the hard part

Everything above is just structure — a handful of small reads. The one genuinely hard thing about HEIC on the web is showing the picture, because that means decoding HEVC, which is patent-encumbered and not shipped in every browser. Safari and recent Chrome/Edge builds decode HEIC natively; Firefox and older browsers do not. An honest tool tries the browser’s own decoder and degrades gracefully:

const img = new Image();
img.onerror = () => showMessage("this browser can't decode HEVC — structure tabs still work");
img.src = URL.createObjectURL(file);   // works in Safari; may fail elsewhere

This is the whole reason so many “HEIC viewers” are really converters: it is easier to transcode to JPG on a server than to decode HEVC in the browser. Reading the structure, by contrast, works everywhere.

Mapping a multi-gigabyte HEIC

Because the entire structure lives in the meta box at the front, the file size is irrelevant to inspection. The recipe is: read a small prefix, find the meta box header, make sure the whole (still small) meta box is buffered, and parse. The coded pixels in mdat are never touched:

let prefix = new Uint8Array(await file.slice(0, 256 * 1024).arrayBuffer());
let { boxes } = walkTopLevel(prefix, file.size);
const meta = boxes.find(b => b.type === 'meta');
if (meta.offset + meta.size > prefix.length)                    // meta runs past the window?
  prefix = new Uint8Array(await file.slice(0, meta.offset + meta.size).arrayBuffer());
const model = parseMeta(prefix, meta);                          // items, props, locations

A HEIF image sequence (a Live Photo, a burst) can be gigabytes; this still opens it on a few kilobytes of reads, because the item list, the dimensions, the HEVC profile and the EXIF all sit before the coded frames.

Where the spec lives

The container is ISO/IEC 14496-12 (ISO Base Media File Format); the HEIF image layer — the meta box, items, properties and references — is ISO/IEC 23008-12; the HEVC coding of the pixels is ISO/IEC 23008-2 (H.265). Apple’s use of it is documented under “HEIF” in the developer libraries. OmniViewer implements the reading side of all three, dependency-free, in the browser.