Proofread a two-page document with a neural model and the full response takes a few seconds. That's fine for a "Check document" button with a spinner. It's unacceptable for an interface that should feel alive — the kind where suggestions start appearing almost immediately, the way they do when checking a single sentence.

The obvious fix is streaming, and the obvious streaming is the one every LLM demo uses: pipe model tokens to the client as they're generated. For proofreading, that's the wrong primitive. This article explains why, and describes the alternative we built for AmberPen: streaming complete, validated edit batches instead of raw generation.

Why token streaming fails for corrections

A proofreading response isn't prose to be typed onto the screen — it's structured data with a validity boundary. An edit is only meaningful whole: range, original text, and replacement together. A token stream hands you fragments:

{"edits":[{"start":17,"end":22,"origi

Nothing here is renderable. Buffer until the JSON parses and you've reinvented the non-streaming response with extra parsing risk. Diff the partially-generated corrected text against the source and it's worse: the diff of an incomplete correction shifts as later tokens arrive, so an underline painted from token 400 might belong somewhere else entirely by token 900. Flickering, migrating underlines are worse than a spinner — they teach users not to trust the interface.

The root issue: a token is a unit of transport. What the UI needs is a unit of meaning. For proofreading, the unit of meaning is a complete edit whose region of the document is final.

The design: diff the stable prefix

AmberPen's correct-mode streaming keeps the request simple — the entire document goes to the model in one call — and moves the cleverness to where the response is processed:

  1. As the model's corrected text streams in, the API continuously diffs the stable prefix of that correction against your submitted text. Incomplete trailing tokens stay buffered; only the portion of the diff that can no longer change is considered settled.
  2. Settled edits — with offsets relative to the full original text — are emitted as complete batches, in document order, with globally sequential IDs.
  3. When generation finishes, a final sentinel closes the stream.

Accumulate every batch and you get exactly the edits a regular request would return. Streaming changes when results arrive, never what they are. That's a property worth insisting on from any streaming API: progressive delivery must not mean approximate results.

What the wire looks like

Request application/x-ndjson with stream: true, and each non-empty line is one complete edit array:

[{"id":0,"start":17,"end":22,"original":"works","replacement":"work"}]
[{"id":1,"start":144,"end":145,"original":"i","replacement":"I"}]
[]

Two protocol details earn their keep:

  • The final empty array [] marks success. Without it, a truncated connection and a finished document-with-no-more-edits are indistinguishable. With it, your client can detect a cut stream and retry instead of silently showing a partial result.
  • Errors after streaming starts arrive as a structured error object, not an edit array and not a bare TCP reset. Mid-stream failures are a fact of networks; they should be representable in the protocol.

With the SDK, none of this is manual — proofreadStream is an async iterator:

const edits = [];
for await (const batch of amberPen.proofreadStream({ text, mode: "correct" })) {
	renderBatch(batch); // show these suggestions now
	edits.push(...batch);
}
// iterator ended ⇔ the final [] arrived ⇔ the result is complete

UI patterns that work (and one that doesn't)

Streaming changes how your interface should manage suggestions:

  • Treat batches as additions to a request-scoped set. New batch, more suggestions. Never mix batches from different document versions — keep the version-discard discipline from the integration guide.
  • Anchor decorations to the original text. Every offset in every batch refers to the document you submitted. That makes ProseMirror-style decorations trivial: paint each batch as it lands.
  • Don't apply edits incrementally. This is the tempting mistake. Applying batch one shifts the text, which invalidates the offsets of batch two. Accumulate, or render as decorations, and apply only on user action.
  • Wait for the sentinel before marking the check complete. "Still checking…" states should track the stream's end, not a timeout.
  • Abort on typing. Pass an AbortSignal; cancellation propagates to the pending model requests instead of letting them burn budget nobody will see.

Progressive explanations, too

Streaming isn't only for the fast path. In evaluate mode, correction proposals stream from the same whole-document request and are evaluated in groups of five — as soon as a group is ready, it goes through the high-reasoning pass and the enriched results stream back: explanation, category, and relevance-ranked replacements. The final group is evaluated even with fewer than five edits, and rejected proposals are dropped, so every batch is safe to accumulate and apply.

The UX payoff is significant for education products: the first explained suggestions appear while the rest of the document is still under review. Feedback starts feeling immediate without sacrificing the quality controls that make it worth reading.

Streaming incremental updates

Streaming and incremental proofreading combine through the SDK: pass a stable textId to proofreadStream, and later checks stream only changed chunks while carrying edits from untouched text into the full-document result. A streamed version enters the cache only after every required chunk completes successfully, so an aborted, failed, or partially consumed check leaves the last complete snapshot intact.

The principle

"Stream tokens" is an answer that leaks the model's internals into your protocol. "Stream the smallest unit your consumer can act on" is an answer designed from the consumer backward — for proofreading, a complete edit with stable offsets into a known document. The same principle travels well: whatever your AI feature returns, stream units of meaning, not units of transport.

You can feel the difference in one request: the streaming guide has the HTTP and SDK details, and a free test key gets a long document streaming edit batches in minutes.

Streaming is one of the properties worth checking before you pick a provider — most proofreading APIs return a single blocking response. The grammar checker API comparison covers which ones stream, which return structured edits, and how they measure up on quality and latency.