What is a .jsonl file? JSON Lines and NDJSON explained

Something handed you a .jsonl (or .ndjson) export, you ran JSON.parse on it, and it said Unexpected non-whitespace character after JSON at position 214. Nothing is broken. JSON Lines is not one JSON document — it is one complete JSON value per line, and that single change is why it can be appended to, split across machines and streamed at 40 million records without ever being held in memory. Here is exactly what is inside one, what goes wrong, and the algorithm that reads a 20 GB one in a browser tab.

What JSON Lines actually is

The whole format is one sentence: a text file where each line is a complete, self-contained JSON value, and the lines are separated by \n. That is it. There is no wrapper array, no commas between records, no schema, no header, no footer, and no way for one record to depend on another.

Compare the two files below. They hold identical data. The one on the left is a JSON document; the one on the right is JSON Lines.

            JSON (one document)                    JSON Lines (three records)

[                                          {"id":1,"name":"Ada"}
  {"id": 1, "name": "Ada"},                {"id":2,"name":"Bob"}
  {"id": 2, "name": "Bob"},                {"id":3,"name":"Cy"}
  {"id": 3, "name": "Cy"}
]

The difference looks cosmetic and is not. To read the left file you must find the closing ], which means reading to the end — the last byte is load-bearing, so a parser has to keep state for the whole document and a truncated file is worthless. To read the right file you need one line. Records 2 and 3 could be on another machine. Record 4 can be appended tomorrow by a process that has never seen the first three, with a plain >>.

JSONL, NDJSON, LDJSON — the same thing with four names

The format was invented independently several times, so the naming is a mess. All of these describe the same bytes:

For MIME types you will see application/x-ndjson (the common one), application/jsonl, and plenty of servers that just say application/json and hope. None of them are registered with IANA. The two spellings that matter in practice are .jsonl and .ndjson, and they are interchangeable — OmniViewer treats /jsonl and .ndjson as the same toolkit for exactly that reason.

A complete file, annotated

Real JSON Lines is almost always events: log lines, analytics, webhook deliveries, model training samples, database rows. Here is a short one with the things that actually turn up in the wild.

{"ts":"2026-03-01T09:14:02Z","event":"session.start","user":{"id":48211,"plan":"pro"}}
{"ts":"2026-03-01T09:14:05Z","event":"search","props":{"q":"invoice 2026","results":17}}
{"ts":"2026-03-01T09:14:11Z","event":"cart.add","items":[{"sku":"WD-1401","qty":1}]}
{"ts":"2026-03-01T09:14:31Z","event":"pay","error":{"code":"card_declined"}}
{"ts":"2026-03-01T09:14:48Z","event":"pay","error":null}
{"ts":"2026-03-01T09:15:00Z","event":"session.start","user":{"id":"anon-7c41"}}
["2026-03-01T09:15:11Z","heartbeat",{"ok":true}]

Four things to notice, because each one is legal and each one surprises somebody:

• Records need not have the same keys. Line 1 has user, line 2 has props, line 3 has items. Nothing enforces a shape. There is no schema in the file, anywhere.
• The same key can hold different types. user.id is a number on line 1 and a string on line 6. Legal JSON Lines; instant rejection by BigQuery, a typed struct, or a Parquet writer.
• null and “missing” are different. Line 4 has an error object, line 5 has an explicit error: null, lines 1–3 have no error key at all. Three distinct states.
• A record does not have to be an object. The last line is an array. Any JSON value is allowed — array, string, number, even a bare null. Most tools assume objects, and most files oblige, but the format does not require it.

That first pair — keys that come and go, types that drift — is the reason the SCHEMA tab exists: it walks every record in the file and tells you that user.email is present in under a quarter of them and that user.id came in two types.

The four rules, precisely

1. One complete JSON value per line. Not a fragment, not a continuation. If a line does not parse on its own, the file is malformed.
2. No unescaped newlines. A JSON string may not contain a literal newline anyway (JSON forbids raw control characters in strings), so a record can always be written on one line — multi-line text just uses \n escapes. This is what makes “split on 0x0A” safe, and it is the property the whole format rests on.
3. UTF-8. Encode as UTF-8. A byte-order mark is not part of any record; a strict reader that does not strip it fails on line 1 with a mystifying error about the invisible U+FEFF.
4. \n is the separator. \r\n is tolerated by every real reader (the \r is trailing whitespace, which JSON ignores), and the final newline is optional — a file may or may not end with one. Blank lines are not records; a tolerant reader skips them, a strict one may complain.

Notice what is not in the rules: no maximum line length, no required key, no ordering, no header. A single record may legitimately be 40 MB long (one enormous embedded document) while its neighbours are 90 bytes.

Why the format exists at all

JSON Lines is what you get when you ask “how do I put JSON in a pipe?” Three properties fall out of the one-value-per-line rule, and every one of them is a thing a wrapped JSON array cannot do:

It is appendable

# add a record to a 400 GB file in constant time
echo '{"ts":"2026-03-01T10:00:00Z","event":"ping"}' >> events.jsonl

# concatenating two exports concatenates the data — no re-parse, no rewrap
cat part-0000.jsonl part-0001.jsonl > all.jsonl

With a JSON array you would have to seek to the final ], back up over it, add a comma, write the record and re-close — and two files cannot be joined with cat at all.

It is splittable

# any line boundary is a safe split point, so N machines can read N shards
split -l 1000000 events.jsonl shard-

# the last 5 records of a huge file, without reading the first 40 million
tail -n 5 events.jsonl

This is the property that made it the default interchange format for distributed systems: Hadoop, Spark, Athena, BigQuery and friends all want to hand byte range k of a file to worker k, and only a line-delimited format lets them find a record boundary without a parser.

It is streamable, with constant memory

A reader holds one record at a time. That is the difference between an export you can process on a laptop and one you cannot, and it is why .jsonl is the format for LLM fine-tuning sets, Elasticsearch bulk bodies, structured application logs, database dumps and anything else that is measured in millions of rows.

What breaks — the mistakes everybody makes

1. A pretty-printed JSON document saved as .jsonl

By far the most common. Someone ran their formatter, or wrote json.dump(data, f, indent=2), and the result is one value spread over 400,000 lines. Every line is a fragment, so every line fails. The giveaway is that line 1 is exactly [ or {.

[                      ← line 1 is not a JSON value, it is the start of one
  {
    "id": 1
  },
  ...

The fix is to re-emit it, one value per line — jq -c '.[]' in.json > out.jsonl does it. (OmniViewer's RECORDS tab detects this case by name and points you at /json instead of listing 400,000 identical errors.)

2. A trailing comma, from thinking in arrays

{"id":1,"name":"Ada"},     ← the comma belongs to an array that isn't there
{"id":2,"name":"Bob"}

Each record is its own document, so there is nothing to separate. The comma makes line 1 “a value followed by junk”.

3. A truncated last line

A process was killed, a disk filled, a download stopped, an S3 multipart upload lost a part. The last record is half a record. This is the failure mode of JSON Lines in production, and it is also the one the format handles most gracefully: records 1 to n−1 are still perfectly readable, which is not true of a truncated JSON array.

4. A byte-order mark

Written by a Windows editor or a .NET StreamWriter default. The three bytes EF BB BF sit in front of record 1 and a strict parser refuses it. Strip the BOM, not the record.

5. Real newlines inside a string

Some naive writer interpolated user text straight into JSON without escaping. Now one logical record spans three lines, and all three are invalid. Legal JSON never has this, which is precisely why a line split is trustworthy.

6. Types that drift between records

"qty": 12 in most records and "qty": "12" in a few, because one producer went through a form. The file parses perfectly and the load into a typed system fails, usually hours later. This one has no error message anywhere in the file — you can only see it by looking at every record, which is what the SCHEMA tab does.

7. JSON.parse on the whole file

Both a correctness bug and a memory bug: it cannot work, and on a 20 GB file it cannot even be attempted — a JavaScript string maxes out around 512 MB in V8, long before the array of parsed objects would.

Reading it in code

Node, streaming, constant memory:

import { createReadStream } from 'node:fs';
import { createInterface } from 'node:readline';

const rl = createInterface({
  input: createReadStream('events.jsonl'),
  crlfDelay: Infinity,          // treat \r\n as one break
});

let n = 0, bad = 0;
for await (const line of rl) {
  if (!line.trim()) continue;   // blank lines are not records
  n++;
  try {
    const record = JSON.parse(line);
    // …do the work, then let it be collected
  } catch (err) {
    bad++;
    console.error(`record ${n}: ${err.message}`);
  }
}
console.log(`${n} records, ${bad} broken`);

Python, same shape:

import json

with open("events.jsonl", encoding="utf-8-sig") as f:   # utf-8-sig eats a BOM
    for n, line in enumerate(f, 1):
        line = line.strip()
        if not line:
            continue
        try:
            record = json.loads(line)
        except json.JSONDecodeError as e:
            print(f"record {n}: {e}")

And the shell, where the format really shines:

jq -c 'select(.event == "pay")' events.jsonl      # filter, stay in JSONL
jq -s '.' events.jsonl > events.json             # JSONL → one JSON array
jq -c '.[]' events.json > events.jsonl           # JSON array → JSONL
wc -l events.jsonl                               # record count (± the last newline)

How we read a 20 GB .jsonl in a browser tab

This is the part that makes JSON Lines interesting to implement rather than just to describe, and it comes down to one observation:

Contrast that with plain JSON, where a chunk boundary can fall inside a string, inside an escape, inside a number, at any depth — and a chunked parser has to carry every possible interpretation forward until one survives. (We wrote that machine too, for fastjsonviewer.com; it has 23 boundary scenarios. JSON Lines needs none of them.)

The scanner

Because of that, the whole engine is a resumable splitter that keeps exactly one thing across a block edge: the bytes of the record it is in the middle of. Here is the core of jsonl-scan.js, simplified only by dropping the statistics bookkeeping:

const LF = 0x0a;

export function createJsonlScanner() {
  const decoder = new TextDecoder('utf-8');
  let bytesSeen = 0;              // absolute offset of the next byte to arrive
  let tail = new Uint8Array(0);   // the record still being assembled
  let tailLen = 0;
  let recordStart = 0;            // absolute offset that record begins at

  function handleRecord(view, start, length) {
    const text = decoder.decode(view).trim();   // also absorbs a CRLF's \r
    if (!text) return;                          // a blank line is not a record
    try {
      const value = JSON.parse(text);
      // …count it, walk its fields, note its type
    } catch (err) {
      // …record `start` — the byte offset to seek to — and the message
    }
  }

  return {
    push(bytes) {                 // ANY slice, in order. 1 byte or 64 MB.
      let from = 0;
      for (let i = 0; i < bytes.length; i++) {
        if (bytes[i] !== LF) continue;
        const slice = bytes.subarray(from, i);
        if (tailLen === 0) {
          handleRecord(slice, recordStart, slice.length);
        } else {
          // the record started in an earlier block: finish it
          tail.set(slice, tailLen);
          handleRecord(tail.subarray(0, tailLen + slice.length), recordStart, tailLen + slice.length);
          tailLen = 0;
        }
        from = i + 1;
        recordStart = bytesSeen + from;
      }
      const rest = bytes.subarray(from);         // carry the partial record
      tail.set(rest, tailLen);                   // (grown as needed)
      tailLen += rest.length;
      bytesSeen += bytes.length;
    },
    finish() {                    // the last record may have no trailing newline
      if (tailLen > 0) handleRecord(tail.subarray(0, tailLen), recordStart, tailLen);
    },
  };
}

Two details worth pointing out. First, recordStart is an absolute offset into the file, carried across blocks — that is what lets the viewer say “record 12,000,412 is broken, at byte 3,918,204,551” and then seek straight to it. Second, there is no streaming TextDecoder here, unlike every other text scanner we ship: a record is always decoded whole, because its boundaries are newlines, so a block edge can never split a character mid-record. One fewer moving part, for free.

The block loop

The worker then just walks the file. File.slice() returns a lazy Blob view, so this reads 4 MB at a time off disk and never materialises the file:

const SCAN_BLOCK = 4 * 1024 * 1024;

const scanner = createJsonlScanner();
let at = 0;
while (at < file.size) {
  const end = Math.min(file.size, at + SCAN_BLOCK);
  const bytes = new Uint8Array(await file.slice(at, end).arrayBuffer());
  scanner.push(bytes);            // the block is now garbage
  at = end;
  postMessage({ type: 'scan-progress', scanned: at, total: file.size });
}
scanner.finish();

What that costs

Peak memory is the honest sum of four things, and none of them scales with the file:

• one 4 MB block — the previous one is collectable the moment push returns;
• one record — the largest single line in the file, whatever that is;
• the field table — capped at 512 distinct paths, a few tens of KB;
• the browsable row window — the first few thousand record offsets and previews, so RECORDS has something to paint.

Everything else is thrown away as it goes. Throughput is bounded by JSON.parse, not by the scan: the byte split runs at GB/s, and parsing typical 200-byte event records lands around 40–80 MB/s in a worker — call it 5–9 minutes for 20 GB, with a progress bar, in a tab, on one core. Every record validated, every field counted, nothing uploaded.

Index, then seek

Browsing is a separate problem from scanning. Keeping an offset for all 200 million records of a 20 GB file would cost more than a gigabyte of typed array, so the viewer keeps offsets for a bounded window and treats opening a record as a seek: one file.slice(offset, offset + length), one JSON.parse, done — the cost of reading record 1,900,000 is the size of record 1,900,000. Broken records found anywhere in the file are carried out of the scan with their offsets attached, so a failure past the window is still one click away.

The RAW and HEX views are windowed by the same principle at a lower level: they read only the bytes needed to paint the screen, which is why they open a 20 GB .jsonl as fast as a 2 KB one. The two conversions — CSV and JSON array — genuinely cannot stream, because their output is one document; those work on a bounded prefix and say so on screen rather than pretending.

JSON Lines compared

The trade is plain: JSON Lines buys append, split and stream with disk space and no types. That is the right trade for a pipeline and the wrong one for storage, which is why so many stacks land the same data as .jsonl and then compact it to Parquet — and why the two mistakes that bite hardest (drifting types, a truncated tail) are both about the moment the JSONL is handed to something stricter.

Open one now

OmniViewer opens every file format in your browser — JSON, JSON Lines, CSV, YAML, XML, Parquet, JavaScript and more — powered by the same windowed engine as fastjsonviewer.com and hugecsv.com. Nothing is uploaded, and it works offline.