Every grammar checker claims quality; a few publish numbers. Those numbers — GLEU, F0.5, precision, recall — come from a specific research tradition with specific blind spots, and understanding them takes about twenty minutes. After this article you'll be able to read any vendor's benchmark table (including ours) and know exactly what's being claimed, what's being omitted, and what to test yourself.

The two questions every evaluation must answer

A grammar corrector can fail in two opposite directions:

  • It misses errors. The sentence "Their happy with the results" comes back unchanged. (Recall failure.)
  • It breaks correct text. The sentence "She leads the CTA working group" comes back with "CTA" changed to "Cat." (Precision failure.)

Users forgive the first and hate the second — an underline that "corrects" a correct word destroys trust faster than a missed typo. That asymmetry runs through everything below, including why the field's standard metric weights the two failures unequally.

There's also a subtler axis: minimal correction vs fluency rewriting. "I have visited Paris last year" can be minimally corrected ("I visited Paris last year") or fluently rewritten ("Last year, I visited Paris"). Both are right; benchmarks disagree about which to reward, and the disagreement is intentional — they were built to measure different things.

GLEU and JFLEG: the fluency measure

JFLEG (JHU Fluency-Extended Grammatical error correction corpus) is a dataset of sentences from English learners' essays, each corrected by four human annotators who were told to make the sentence fluent — not merely error-free. That instruction matters: JFLEG references rewrite freely, so it rewards systems willing to make broader, more natural corrections.

Its metric, GLEU, adapts the BLEU score from machine translation. Where BLEU rewards matching reference n-grams wholesale, GLEU is correction-aware: it rewards n-grams your system changed when the references changed them too, and penalizes n-grams your system kept when the references changed them. Scores run from 0 to 1; on the JFLEG development split (754 sentences, all four references), published product scores in our comparison ran from 0.31 to 0.56.

GLEU's known weakness is the flip side of n-gram matching: a correction that's valid but unlike all four references gets little credit. Fluency has many right answers, and no metric knows all of them. Treat GLEU as a strong signal of rewriting ability, not a truth oracle.

ERRANT and F0.5: the precision measure

ERRANT (ERRor ANnotation Toolkit) works at the level of individual edits. Given a source sentence and a system's output, it extracts the edit script — insert this, delete that, replace this with that — and classifies each edit into a fine-grained type system (R:VERB:TENSE, M:DET, U:PREP, PUNCT, SPELL, and dozens more). Comparing those extracted edits against gold annotations yields the classic trio:

  • Precision — of the edits the system made, how many matched a human annotation?
  • Recall — of the errors humans marked, how many did the system fix?
  • F0.5 — their weighted harmonic mean, counting precision twice as heavily as recall. The field chose this deliberately: false corrections are roughly twice as damaging as missed ones in user-facing tools.

CWEB is the dataset that makes ERRANT scores interesting for product evaluation. It combines two test sets of genuine web text (CWEB-G and CWEB-S, 6,845 sentences in our runs) where error density is low — most sentences are already correct. That makes it a false-positive trap: an over-eager corrector racks up edits, but each unmatched edit sinks its precision. In our CWEB runs, edit counts ranged from 710 (LanguageTool, ultra-conservative) to nearly 6,000, and precision ranged from 5.8% to 40.8% — the dataset pulls systems apart exactly where users are most sensitive.

How to read a vendor benchmark table skeptically

Armed with the metrics, here's the checklist. Every item is a real way benchmark claims go wrong:

  1. Which split? Development splits are where everyone tunes; test splits are where honesty lives. "State of the art" on a dev set after prompt-tuning against it is overfitting with extra steps. (Our rule, documented in the benchmark methodology, is simple: never tune prompts against a held-out test split.)
  2. Which scorer, which version? ERRANT and GLEU have official implementations and versions; scores from a vendor's custom scorer aren't comparable to anyone else's.
  3. Did every product get identical inputs? Same sentences, same formatting, scored locally with the same tools. Comparing a number you computed against a number from a competitor's marketing page is not a benchmark.
  4. What's missing? A table with recall but no precision is hiding false positives. Quality with no latency is hiding the seconds. English-only numbers say nothing about multilingual claims.
  5. Does the corpus resemble your text? Learner essays (JFLEG) and clean web prose (CWEB) measure complementary behaviors. Neither is your support inbox, your students' drafts, or your users' legal memos.

A worked example: reading our own table

Here's the honest reading of the published comparison, including the parts we don't win:

Product JFLEG GLEU CWEB precision CWEB recall CWEB F0.5 p50
GrammarBot Neural 0.413 40.8% 62.5% 43.9 1,000 ms
AmberPen 0.561 37.4% 65.5% 40.9 217 ms
LanguageTool 0.307 37.8% 18.3% 31.2 287 ms
Sapling 0.470 23.6% 49.1% 26.3 620 ms

GrammarBot Neural wins CWEB F0.5 because precision dominates that metric: on mostly-correct web text it proposes fewer unmatched edits. AmberPen wins JFLEG by a wide margin because its fluency rewriting is stronger, and wins recall by finding more of the real errors — while responding 4.6× faster at the median. LanguageTool's 18% recall is what a rule-based ceiling looks like in numbers.

Which product is "best"? On this table, it depends on your text and your tolerance for false positives versus missed errors. That is the lesson: any vendor whose benchmark section has a single trophy number is selling, not measuring.

Beyond reference metrics

Reference-based scores share one more blind spot worth knowing: a correction can be valid yet match no reference, so every system is scored slightly below its true quality. A rigorous evaluation therefore adds reference-free checks. Our internal suite includes:

  • No-op accuracy — feed the system already-correct text and count unnecessary edits per 1,000 characters. The most user-visible failure of all.
  • Semantic preservation — does the corrected text still mean what the writer meant?
  • Formatting round-trips — Markdown, whitespace, emoji, and Unicode must survive intact.
  • Latency percentiles — p50, p95, and p99, because the median is the experience and the tail is the complaint.
  • Breakdowns — by error type, input length, and language, since averages hide where a system is weak.

If you're evaluating tools for a product, build a small private set from your own real text — a few hundred representative sentences — and score every candidate against it. Public benchmarks narrow the field; your corpus picks the winner.

The takeaway

GLEU-on-JFLEG measures fluent rewriting; ERRANT-on-CWEB measures disciplined correction. Precision failures and recall failures are not symmetric, which is why F0.5 exists. And every number is only as trustworthy as the methodology page behind it — splits, scorers, identical inputs, and no tuning on the test set.

You'll find all of that for our numbers on the benchmarks page, and the per-product breakdowns on our comparison pages for AmberPen vs LanguageTool, AmberPen vs Harper, AmberPen vs Sapling, and AmberPen vs GrammarBot Neural. Then run your own text through a free test key — the benchmark that matters most is the one you build yourself.

Where these numbers get used

Scores are only useful once they inform a decision. These articles apply the metrics above to the choices teams actually face:

Choosing a tool

Building the integration