Teams tend to discover their annotation quality ceiling the same way: accuracy climbs steadily, plateaus somewhere around 92%, and then refuses to move no matter how many review passes get added. Adding reviewers at that point mostly buys you more confident disagreement.
Measure agreement before you measure accuracy
Inter-annotator agreement tells you something accuracy cannot: whether the task is well-defined. If two trained annotators looking at the same item reach different labels, the problem is not in either of them. It is in the guidelines.
We run agreement scoring from the first batch onward, on a deliberate overlap sample. Where agreement is low, we pull the disputed items into a review session and almost always find a category boundary nobody had written down.
Edge cases belong in the spec, not in the reviewer's head
The most common failure is a guideline document that describes the clean cases beautifully and says nothing about the messy ones. Annotators then invent consistent-but-divergent rules, and those rules propagate into the dataset as systematic noise the model will happily learn.
The fix is boring: every disputed item that reaches review becomes a worked example in the guidelines, with the reasoning attached. A mature spec on a complex task is mostly edge cases, and that is a sign of health rather than decay.
Gold standards catch drift that review misses
Seed a small set of items with known-correct labels into every batch, unannounced. Drift shows up in the gold set weeks before it shows up in model performance, and it tells you which annotator, which shift, or which category is sliding.
What good looks like
On production datasets we target 98% or better, and we get there through specification work rather than through volume of review. The teams that struggle are rarely short of reviewers. They are short of an agreed definition of the thing they are labelling.