Writing

The Model Who Cried Wolf

A journey of discovery to train a small local proofreader that fixes spelling and grammar, explains every fix like an editor, and leaves what's already right alone.

#Thought

Most of a proofreader's mistakes aren't missed errors. They're edits to text that were already fine.

A missed comma costs nothing as typically nobody notices it. A wrong edit costs attention. You have to stop, read the suggestion, decide it's wrong and dismiss it.

Do that a few times and you stop reading the suggestions, which means the tool has stopped working even when it's right. This is true of many things in software engineering, and it always reminds of the famous story of “The Boy Who Cried Wolf!”.

So when I set out on a journey to build Sherlock, a small writing model that fixes spelling, grammar and punctuation but can also explain these edits, I put three questions ahead of all the others.

Can a model be trained better to leave already correct text alone?
Can we run locally and provide real-time feedback as you type?
Can we explain edits to make Sherlock sound like a human editor?

This post discovers how I trained the initial version of Sherlock, what went wrong along the way, and what the numbers currently look like.

Fun fact:
Sherlock is named after our young Boston Terrier who is currently in training. Still undecided on whether the model is easier train.

#Try

It’s still early days for Sherlock. I’m sharing it in preview while I learn what works, what misses the mark, and how to make the results more useful.

Deployed for real, it will sleep when nobody’s playing with it (unlike our Boston Terrier). The first run after a quiet moment may take a little longer while it schedules a GPU for use.

Try SherlockPreview

Examples

Run the model to see results

#Problem

The obvious way to build a proofreader in 2026 is to simply prompt a general language model with "fix the errors in this paragraph" (make no mistakes anyone!?). It works, sort of. The trouble shows up when you hand it a paragraph with nothing wrong in it.

The additional element is that I wanted to also train the model to give contextual feedback of the edits it makes. Trying to make the edits feel more like a human editor rather than traditional rule based approaches.

I took 1,000 real paragraphs from Wikipedia, each one already corrected by a human editor, and asked models to proofread them. Each general model was given a prompt written for the job (fix spelling, grammar and punctuation, leave correct text alone, answer with only the fixes) and three examples, one of them a paragraph with nothing to fix.

I also marked more than 800 real Wikipedia fixes by hand as either a real error or only a style change. Whenever Sherlock got something wrong in my own use, I fixed that whole mistake in the training data.

The correct answer for almost every one of these is "nothing to fix".

  • Qwen 3.5 4B changed 63%
  • Gemma 4 26B changed 53%
  • Gemma 4 E4B changed 43%
  • Sherlock currently changes 25%

This interactive scene needs WebGPU or WebGL2.

Show
  • Wrong edit
  • Real fix the editor missed
  • Source already broken
  • Left alone
1,000 paragraphs a human editor had already corrected. Every change by every system was read blind under the same rule.

Changing a paragraph isn't always a mistake. Editors leave real errors behind. So every change a model made was read under one strict rule:

A change that fixes no real error counts as wrong, even if the result still reads fine.

The readers were language models working blind from that rule. They weren't told whose edits they were reading, and everyone's changes were shuffled together.

Of Sherlock's 251 changes, 116 fix real errors the editor had left behind. 119 contain a wrong edit. 16 fall on text that was already broken (stripped templates, glued references), where nobody can say what's right.

Of 1,000 finished paragraphs

Changed

Got a wrong edit

Got a real fix the editor missed

Sherlock

25.1%

11.9%

11.6%

Gemma 4 26B

53.2%

24.1%

25.7%

Gemma 4 E4B

43.3%

29.3%

10.6%

Qwen 3.5 4B

63.1%

46.6%

13.2%

Against the two models its own size, Sherlock currently makes less than half the wrong edits (a quarter of Qwen's) and finds about as many of the real ones.

The honest point against Sherlock is Gemma 4 26B, six times Sherlock's size. It finds more than twice as many of the errors the editor missed. It also puts a wrong edit into one finished paragraph in four to do it, twice Sherlock's rate. Which side of that trade you want depends on whether you'd rather miss an error or dismiss a bad suggestion.

General models aren't stupid. They're trained to be helpful, and an edit looks like help. On sentences built to tempt an edit (a correct subjunctive, a correct "neither … nor", a dangling-looking modifier that isn't one), same-size prompted models edit 7–17% of them. Sherlock edits 0.8%: 11 of 1,332, and it still finds 72% of the real errors planted among them. Gemma 4 26B edits 2.8% and finds more of the planted errors (84%). On clean, built sentences, the bigger model is the stronger grammarian.

#Precision

There's a way to help over-eager models, ask several times and keep only what they agree on. Sample a model eight times, and keep an edit only if seven or eight of the samples make it. That works. Qwen 3.5 4B with 8-sample voting drops from changing 63% of correct paragraphs to 6–11%.

The trade of here is that you pay for it on every request, eight samples instead of one.

Sherlock checks a paragraph that needs zero edits in about 100 ms. With eight-sample voting, the same check on the same DGX Spark takes 1,400–1,500 ms, roughly 14–15× as long.

I wanted the opposite trade when creating Sherlock in that you pay once, in training, and then run one fast pass forever.

#Model

Sherlock starts from NVIDIA Nemotron 3.5 Bolt E4B, a small model with about 4B effective parameters. I simply train a LoRA adapter on it, merge it, and quantise to 4-bit (Q4_K_M, 5.3 GB), so it runs in llama.cpp on ordinary hardware. It can even run super fast locally on my MacBook Pro M4 Max.

It does one job. It doesn't chat, summarise or rewrite your paragraph in a nicer voice. It gets a document as blocks and answers with a list of fixes in a bespoke compact format.

Most paragraphs need nothing, so the most common answer costs about as little as an answer can. Across 3,000 real paragraphs, Sherlock writes about 8 output tokens on average (median: 2). General models given the same kind of bare format write 14 – 27 asked once, mostly because they change more paragraphs. With voting, they write 107 – 227.

This interactive scene needs WebGPU or WebGL2.

Sherlock8.2 tokens (median 2)Gemma 4 E4B, asked once13.8 tokens (median 10)Gemma 4 26B, asked once17.4 tokens (median 11)Qwen 3.5 4B, asked once26.8 tokens (median 13)Gemma 4 E4B, 8-sample voting107 tokens (median 82)Gemma 4 26B, 8-sample voting141.5 tokens (median 94)Qwen 3.5 4B, 8-sample voting227.1 tokens (median 126)
One falling glyph per token written, same 3,000 paragraphs, same speed.

Additionally you can ask Sherlock why an edit was made, and it tells you. Open a fix and it explains it in one sentence about your exact words:

Each of the volunteers has brought their own lunch. The weather effected → affected the turnout less than we feared.


Sherlock: "Affected means the weather changed the turnout, while effected would wrongly suggest the weather caused something physical."

The note is never part of the proofreading answer. It's a separate request, made only when you open a fix, so the common answer stays tiny and you only pay for the explanations you actually read, about half a second each. It's the same model, without needing a separate one with around 44,000 of its training examples teach it to explain a fix.

#Training

The vast majority of the training data started as brand new writing. Large open models I ran myself write texts in the voice of described writer personas, complete with the slips that writer would hopefully make, and a judge throws out any that don't read like a real person (about 9,400 of them). The fixes come from two different models correcting the same text independently. If they agree on an edit then it is kept.

An edit only one makes survives only if a separate check agrees the original really is wrong at that spot. Rare grammar is written on purpose, correct sentences containing a subjunctive or a "neither … nor", so an error can be planted with an exact answer. Fixes are paired with twins, the same kind of sentence already correct and left alone, so restraint comes from the data rather than from an instruction.

Source

Meandhimwasgoingtothelibaryontuesday,weatheritrainedornot.

Teacher A

He and IweregoingtothelibraryonTuesday,whetheritrainedornot.

Teacher B

MeandhimwereheadingtothelibraryonTuesday,whether or not it rained.

Gold

He and IweregoingtothelibraryonTuesday,whetheritrainedornot.

Both teachers' shared fixes fuse into gold; one-teacher fixes are verified; overlaps keep one whole span.

Diagram: Two teachers

I won’t share the full recipe, but a high level breakdown of training and costs are as follows:

Training examples

485,901

Source data

Commercially licensed sources only, or written for the job by open models I ran myself. Never scraped essays, zero benchmark data.

Training

One pass over the data at around 10.5 hours on one rented GPU (B200)

Total spend

About $224, data and training together. The training run itself was about $89.

Other

Benchmarks, testing and serving run on a DGX Spark on my desk

#Speed

Same GPU (a single NVIDIA DGX Spark), same 300 real paragraphs, one request at a time, everything streamed, every model on its own with the same prompt as above:

This interactive scene needs WebGPU or WebGL2.

Paragraph sentAnswer doneSherlockfirst fix 224 ms, whole 253 mswrong edit on 12% of finished text7Qwen 3.5 4B, asked oncefirst fix 234 ms, whole 267 mswrong edit on 47% of finished text6Gemma 4 E4B, asked oncefirst fix 379 ms, whole 480 mswrong edit on 29% of finished text3Gemma 4 26B, asked oncefirst fix 380 ms, whole 471 mswrong edit on 24% of finished text3Gemma 4 E4B, 8-sample voting (7 of 8)first fix 824 ms, whole 995 mswrong edit on 9% of finished text1Qwen 3.5 4B, 8-sample voting (7 of 8)first fix 1,047 ms, whole 1,278 mswrong edit on 3% of finished text1Gemma 4 26B, 8-sample voting (7 of 8)first fix 1,602 ms, whole 1,798 mswrong edit on 18% of finished text1
Show
Same GPU, same 300 paragraphs, one at a time, every model bare with a prompt written for the job. First heat: paragraphs each model fixed; the flash is the first fix on screen, the pulse ends when the whole answer is in. Second heat: paragraphs it left alone, no flash. Beside each lane: how often it edits text that was already right.

First fix on screen (median)

Whole answer (median)

Nothing to fix (median)

Paragraphs it fixed

Correct changes

Sherlock

224 ms

253 ms

101 ms

149

67%

Qwen 3.5 4B

234 ms

267 ms

190 ms

212

45%

Gemma 4 E4B

379 ms

480 ms

479 ms

218

55%

Gemma 4 26B, asked once

380 ms

471 ms

274 ms

203

72%

Gemma 4 E4B, 8 samples

824 ms

995 ms

1,373 ms

59

76%

Qwen 3.5 4B, 8 samples

1,047 ms

1,278 ms

1,545 ms

40

89%

Gemma 4 26B, 8 samples

1,602 ms

1,798 ms

1,542 ms

133

78%

“Correct changes” is measured on all 3,000 real paragraphs, not just the 300 for request time comparison.

As fast as Qwen 3.5 4B asked once, but two in three of Sherlock's changes are right, against fewer than half of Qwen's.

Asked once with a lean prompt, a general model is quick too. Qwen gets its first fix on screen about as fast as Sherlock, and the Gemmas take under twice as long. First fix and whole answer are both timed over the paragraphs each model fixed; on a paragraph that needs nothing, Sherlock answers in about a tenth of a second and Qwen in about two tenths. Voting can't show a fix until seven samples have written it, so against voting, Sherlock's first fix and whole answer both arrive 4–7 times sooner, and 14–15 times sooner when there's nothing to fix. So speed alone isn't the case for training. The case is speed with restraint.

Asked once, the general models are fast but make two to four times the wrong edits. Voting brings their wrong edits down, but then they're the slow ones.

#Keep Bias

To be a little more efficient whilst developing Sherlock I added a setting that makes it more reluctant to start a fix. It doesn't change what a fix says, only whether one starts, and it costs nothing per request. I call it the keep bias.

Turning it up trades fixes for restraint. Here it is against same-size models voting eight times, on the same 3,000 real paragraphs, with every change on the finished paragraphs read under the same strict rule:

Finished paragraphs changed

…with a wrong edit

Editors' fixes found (general / grammar)

Output tokens per text

Sherlock, bias off

25.1%

11.9%

26.8% / 35.1%

8

Gemma 4 E4B, 7 of 8 agree

20.3%

9.4%

23.4% / 27.7%

107

Gemma 4 E4B, 8 of 8 agree

14.9%

6.5%

20.8% / 23.6%

107

Sherlock, bias 1.5

15.3%

5.7%

23.7% / 28.9%

6

Qwen 3.5 4B, 7 of 8 agree

11.1%

3.0%

19.8% / 18.0%

227

Sherlock, bias 3

7.6%

2.3%

19.6% / 21.5%

4.5

Qwen 3.5 4B, 8 of 8 agree

5.7%

1.3%

15.4% / 11.4%

227

Sherlock, bias 4.5

4.2%

0.7%

14.8% / 13.6%

3.5

At each level of wrong edits, Sherlock with the dial turned up finds about as many of the editors' general fixes as a same-size model voting eight times, and more of the grammar fixes.

It writes 18–65 times less output to get there. Gemma 4 26B voting sits somewhere else: 14–18% wrong, but finding 37–39% / 50–52% of the editors' fixes. Sherlock's keep bias only turns towards caution, so it can't get there.

A live version I have runs with keep bias off. For now it sits within a bespoke WebGL text editor (named Caret) where you see every suggestion and can dismiss them. Testing internally more edits is worth a few more dismissals to really see how the user experience would feel to find the right balance.

#Imperfections

Reading every wrong edit on real text, the biggest kinds are:

  • Introducing an error into correct text (37 of 119)
    an inserted word that breaks a list, for example.
  • Style changes that fix nothing (26)
  • Changing a correct name or specialist term (24)
    rare words "corrected" into familiar ones, a stylised lower-case title capitalised.
  • Repairs of already-broken text that leave it broken (18)

The keep bias removes most of them first: at bias 3, about 23 of the 119 remain.

Other honest limits

Right now Sherlock is English only (British and American), and the benchmarks used are mine (frozen before training, but mine none the less). A bigger general model finds more of what editors miss. An error surrounded by clean sentences is found less often than the same error on its own.

#Retrospective

  • Measure correct text first.
    If your benchmark is all errors, a model that edits everything looks good.
  • Count a wrong edit as a cost.
    A change that fixes nothing isn't a style choice. It's a mistake the writer has to dismiss.
  • Look for what your data never shows.
    The model's blind spots were exactly the things it had never seen.
  • Make the empty answer cheap.
    Most text is fine, so most answers should cost almost nothing.

#Conclusion

Sherlock is still in its very early stages, but if you give Sherlock a thousand paragraphs that are already right, it leaves three quarters of them alone. Models its own size make two and a half to four times as many wrong edits. On sentences built to bait an edit, they take the bait 8 to 20 times as often.

To match its restraint, they have to vote eight times. Even then, Sherlock's first fix lands 4 to 5 times sooner, its whole answer 4 to 5 times sooner (14 to 15 times when there's nothing to fix), and it writes a fraction of the output.

A model that needs three times its memory catches more of what editors miss, and pays for it with twice the wrong edits on finished paragraphs. But a proofreader you've trained to ignore finds nothing at all.

Sherlock is the one trying to keep you reading and in flow state. It is cheap to train, a tenth to a quarter of a second per paragraph, small enough and fast enough to run on a laptop.

Home/
Writing/writing
Models/models
Events/events
About/about
Web/topics/web
AI/topics/ai
Fine-tuning/topics/fine-tuning
Training/topics/training
Product/topics/product
Engineering/topics/engineering
Lighttheme
Darktheme
Match systemtheme
RSS feed/rss.xml
Subscribe to events calendar/events.ics
Copy link to this pageaction