Beyond claims data: How to train trustworthy coding NLP

Overview

Accuracy is the backbone of medical coding, yet many organizations still rely on claims data as their primary source of truth when training natural language processing (NLP) engines. For coding professionals, this raises an important question: Can systems built on unaudited, speed‑driven claims really reflect expert‑level coding?

This article explores why claims data falls short, how coder bias shifts in accuracy‑driven environments, and what it truly takes to create a gold‑standard corpus capable of training high‑performing NLP systems.

The limits of blind coding and claims-based accuracy

In medical coding, we talk about bias most often in settings where coders are expected to work independently and quickly – because speed is tied directly to reimbursement timelines. Under these pressures, bias doesn’t just distort performance; it shapes it. It can artificially inflate accuracy metrics, reinforce habitual shortcuts and reward speed over precision.

But when the goal shifts to identifying the most correct answer, everything changes.

In an accuracy‑first environment, coders should have full access to patient history, documentation context and clinical nuance. This context is essential for building a gold standard. Yet blind‑coding studies – often cited to argue that gold‑standard creation is impossible – tell an incomplete story. There are studies that show coder agreement below 50%, but they examine coders working in isolation, without collaboration.
 

Consensus coding: When accuracy becomes achievable

Consensus‑driven review reframes the debate. When expert coders examine documentation together, discuss rationale and validate decisions, agreement stops being an exception and becomes the expected outcome.

This is how you build a corpus designed for:

• Accuracy, not speed
• Clarity, not convenience
• Reliability, not throughput

And it’s the only kind of dataset that can meaningfully train high‑quality NLP.
 

Why relying on claims data for NLP training falls short

Despite known limitations, many organizations continue to use claims data as the primary feedback loop for NLP training. Claims can be useful in one scenario: if an organization wants the NLP engine to mimic how their internal team codes. In those cases, claims feedback reflects local habits and preferences.

But if the goal is to build NLP that follows coding guidelines, clinical logic and industry expectations, claims data becomes problematic.
 

The central question: Who coded the claim?

In many workflows:

• Some claims are coded by certified professionals
• Others by staff with limited coding exposure
• Sometimes clinicians code the claims themselves

Even certified coders may be producing data that is:

• Unaudited
• Speed‑driven
• Influenced by payer edits, not coding guidelines

This contradicts industry norms. AHIMA and AAPC expect coders to:

• Maintain 95%+ accuracy
• Complete ongoing continuing education
• Undergo routine audits, sometimes weekly if accuracy decreases

Yet many provider organizations do not apply this rigor to their claims teams. This raises critical questions for automation:

• Should an NLP model be trained on data that would not meet human‑coder auditing standards?
• Is it acceptable for automation to be held to a lower accuracy bar than human coders?
• If industry groups insist on routine quality checks, why would we skip that step when training machines?

The answer is clear: We shouldn’t. NLP deserves the same level of scrutiny as any human coder – if not more.
 

What a true gold standard requires

If we want high‑quality NLP systems – systems that coding experts trust – then developers must take responsibility for building a rigorous, validated gold standard. That means holding automation to the same expectations we place on people.
 

The gold standard corpus: What it must contain

A true gold‑standard dataset must be:

• Consensus‑reviewed by expert coders
• Guideline‑aligned, reflecting ICD‑10‑CM/PCS, CPT® and payer rules
• Fully annotated, with explicit rationale for every code
• Continuously expanding, covering all code sets and customer scenarios
• Audited regularly, just like human coders

This corpus should power:

• NLP model refinement
• Machine learning development
• Ongoing quality assurance

A structured approach to building it

A scalable gold‑standard workflow typically includes:

1. Claims feedback – Useful for identifying discrepancies and customer‑specific coding behaviors
2. Expert review – Ensures the feedback aligns with coding guidelines and clinical truth
3. Consensus validation – The final step that elevates data from “reviewed” to gold standard

This tiered model ensures NLP learns from the best human coders – not the fastest, not the most overloaded and not the least reviewed.
 

Conclusion: Accuracy must be intentional

Gold‑standard development isn’t a “nice to have” – it’s the foundation for building accurate, trustworthy automated coding systems. Claims data alone cannot meet this bar, and it was never designed to.

If the industry wants automation that reflects the judgment of its most skilled coders, then developers must create the conditions those coders rely on: context, rigor, collaboration and continuous auditing.

That is how we build NLP systems that support coding teams, strengthen documentation integrity, and help enable safer, smarter more accurate healthcare.

April Russell, MBA, CCDS-O, CPC, COC, CPC-P CRC, is an NLU content manager at Solventum.