Inside a JPEG: markers, EXIF & the quantization tables

A .jpg file is the most common photo on Earth, and its container is unusually approachable: a stream of markers — a 0xFF byte and a code — where a handful of small segments at the front hold everything a tool needs, and the compressed pixels trail behind them. This is a tour of that layout: the FF D8 that opens every JPEG, the APP segments where EXIF and the GPS position hide, the quantization tables that encode the save quality, and the difference between baseline and progressive. Along the way we’ll write the JavaScript to strip metadata without re-encoding a pixel and to map a huge photo from just its header markers.

Everything is a marker

JPEG doesn’t have a header struct the way PNG or GIF does. Instead the whole file is a sequence of markers. A marker is a 0xFF byte followed by a non-zero code byte: FF D8 is start of image (SOI), FF D9 is end of image (EOI). Most markers introduce a segment whose next two bytes are a big-endian length — and that length includes the two length bytes themselves, a detail worth pinning down because it is the classic off-by-two:

FF E1 00 88  ...        ← APP1 marker, segment length 0x0088 = 136 bytes
             └─ payload is 136 − 2 = 134 bytes

A few markers stand alone with no length or payload at all: SOI (FF D8), EOI (FF D9), the restart markers RST0–RST7 that punctuate the scan, and TEM. Everything else is a length-prefixed segment you can skip over in one jump.

The segments a photo is made of

A typical camera JPEG, in file order, looks like this:

The crucial observation for a viewer: every segment that describes the image — its size, its quality, its metadata — appears before the first SOS. The scan data after SOS is the bulk of the file, and, like PNG’s IDAT or MP4’s mdat, you never have to read it to understand the photo.

Walking the markers

The walk is a tight loop: find the 0xFF, read the code, and either step over a standalone marker or read the two-byte length and jump the whole segment. It stops the moment it sees SOS:

function* walkSegments(bytes) {
  if (bytes[0] !== 0xff || bytes[1] !== 0xd8) throw new Error('not a JPEG');
  let off = 2;
  while (off + 2 <= bytes.length) {
    if (bytes[off] !== 0xff) break;          // corrupt: not on a marker
    const code = bytes[off + 1];
    if (code === 0xff) { off++; continue; }   // fill bytes before a marker
    // Standalone markers carry no length.
    if (code === 0xd8 || code === 0xd9 || code === 0x01 ||
        (code >= 0xd0 && code <= 0xd7)) {
      yield { code, off, size: 2 };
      off += 2;
      continue;
    }
    const len = (bytes[off + 2] << 8) | bytes[off + 3];   // includes the 2 len bytes
    const payload = bytes.subarray(off + 4, off + 2 + len);
    yield { code, off, size: 2 + len, payload };
    if (code === 0xda) return;               // SOS: the scan follows — we're done
    off += 2 + len;
  }
}

Because every segment announces its own length, this reads only the small stuff and lands on SOS after a handful of iterations, no matter how large the photo is.

Where EXIF — and your location — live

The APP1 segment is the one that matters for privacy. Its payload starts with the ASCII tag Exif\0\0, and what follows is a complete TIFF structure: a byte-order mark (II little-endian or MM big-endian), the magic number 42, and an offset to the first image file directory (IFD). Each IFD is a list of 12-byte entries — tag, type, count, and either an inline value or an offset to one:

const le = data[0] === 0x49;                 // 'II'
const u16 = (o) => le ? data[o] | data[o+1]<<8 : data[o]<<8 | data[o+1];
const u32 = (o) => le ? (data[o] | data[o+1]<<8 | data[o+2]<<16 | data[o+3]*2**24)
                      : (data[o]*2**24 | data[o+1]<<16 | data[o+2]<<8 | data[o+3]);

function readIFD(at) {
  const n = u16(at), out = [];
  for (let i = 0; i < n; i++) {
    const e = at + 2 + i * 12;
    out.push({ tag: u16(e), type: u16(e + 2), count: u32(e + 4), valueAt: e + 8 });
  }
  return out;                                 // last 4 bytes after: next-IFD offset
}

IFD0 holds the camera Make (tag 0x010F), Model (0x0110), Software and DateTime. Two entries are pointers to sub-directories: the Exif IFD (0x8769, with exposure and lens data) and the GPS IFD (0x8825). Inside the GPS IFD, latitude and longitude are stored as three RATIONALs — degrees, minutes, seconds — plus a reference of N/S and E/W:

// A GPS coordinate is deg + min/60 + sec/3600, signed by its ref.
const dms = (e) => rational(e, 0) + rational(e, 1) / 60 + rational(e, 2) / 3600;
const lat = (latRef === 'S' ? -1 : 1) * dms(latitudeEntry);
const lon = (lonRef === 'W' ? -1 : 1) * dms(longitudeEntry);
// → +48.8577, +2.2950  — the exact spot the photo was taken

Quality is a table, not a number

When you “save at quality 85”, the encoder doesn’t write an 85 anywhere. It uses that number to scale a pair of quantization tables — one for luma, one for chroma — and stores those in the DQT segment. Each table is 64 values, one per frequency coefficient; the encoder divides the discrete-cosine-transformed block by them and rounds, and larger divisors throw away more high-frequency detail. Higher quality means smaller divisors.

Since the standard IJG scaling is a known formula, you can run it backwards to recover an approximate quality. Compare the stored luma table against the standard one (Annex K), take the mean ratio, and invert the scale:

// STD_LUMINANCE = the Annex K.1 base table (natural order);
// ZIGZAG maps a stored (zig-zag) index to its natural position.
function estimateQuality(zigzagTable) {
  let sum = 0, n = 0;
  for (let k = 0; k < 64; k++) {
    const q = zigzagTable[k], base = STD_LUMINANCE[ZIGZAG[k]];
    if (q && base) { sum += (q * 100) / base; n++; }
  }
  const scale = sum / n;
  return scale <= 100 ? Math.round((200 - scale) / 2)   // Q ≥ 50
                      : Math.round(5000 / scale);        // Q < 50
}

It’s an estimate — an encoder that ships custom tables (many phones and Photoshop do) will read a little differently — which is why the viewer labels it “≈”. But for the vast majority of files it lands within a point or two of the setting used.

Baseline, progressive and subsampling

The frame marker tells you how the image decodes. SOF0 is a baseline JPEG that paints top to bottom in one pass; SOF2 is progressive, storing the image in successive refinement passes so it sharpens as it loads. The frame header also lists the components and their sampling factors, which give the familiar chroma-subsampling shorthand:

const y = components[0];                      // the luma component
if (y.h === 1 && y.v === 1) return '4:4:4';   // full chroma, no subsampling
if (y.h === 2 && y.v === 1) return '4:2:2';   // half horizontal
if (y.h === 2 && y.v === 2) return '4:2:0';   // half both ways — the common one

4:2:0 halves the chroma resolution in both directions — invisible on photos, visible on sharp coloured text — and is why a screenshot of code often looks muddy as a JPEG.

Stripping metadata without touching a pixel

Here is the payoff of the marker structure. To remove EXIF, XMP and comments, you don’t decode or re-encode anything — you copy the file and drop the metadata segments. The compressed scan is untouched, so the result is pixel-identical. You never even need the whole file in memory: emit a plan of “keep these byte ranges” and let the platform stream it:

function planClean(segments, scanStart, fileSize) {
  const parts = [{ start: 0, end: 2 }];        // SOI
  for (const s of segments) {
    if (isMetadata(s)) continue;               // drop EXIF / XMP / COM / vendor APPn
    parts.push({ start: s.off, end: s.off + s.size });   // keep DQT, SOFn, DHT…
  }
  parts.push({ start: scanStart, end: fileSize });        // the whole scan + EOI
  return parts;                                // → new Blob(parts.map(sliceFile))
}

Because the scan is copied as one slice of the original file, the browser streams it straight to disk — a multi-gigabyte photo is cleaned without its pixels ever being read into memory.

Mapping a photo of any size

Put the pieces together and the whole toolkit costs almost nothing, regardless of file size. The metadata segments cluster in the first few kilobytes, so a buffered reader serves the entire marker walk from one small window; the walk stops at SOS; and the scan is only ever referenced by byte range, never materialised. Dimensions, quality, subsampling, EXIF and the segment map all come from that tiny prefix. The only operation that needs the pixels is the on-screen preview — and that’s handed to the browser’s own decoder, which memory-maps as it pleases.

Where the spec lives

The container is ITU-T T.81 (ISO/IEC 10918-1), the original 1992 JPEG standard; the JFIF APP0 header and the EXIF APP1 structure are separate specifications layered on top (EXIF is TIFF/6.0 tags in a JPEG segment). The quantization and Huffman tables in the callouts above are the ones in Annex K. None of it requires a library to read — a few hundred lines of plain JavaScript walk the markers, decode the TIFF and recover the quality, which is exactly what the OmniViewer JPEG toolkit does, entirely in your browser.