Case Study: Empowering Domain Experts to Make Clinical Judgment Explicit

The Challenge: Judgment-Based Variables Require More Than Automation

‍

In oncology research and registries, one of the most fundamental variables is also one of the hardest to define consistently:

Does this patient have evidence of cancer?

Answering that question requires clinical interpretation, not keyword matching. Notes often include:

• Ambiguous or equivocal language
• Statements of intent rather than diagnosis
• Evolving impressions across visits
• Explicit negation, uncertainty, or omission

Historically, ensuring consistency has required extensive abstraction manuals, long training periods, and ongoing adjudication. Alternatively, institutions could build a custom abstraction pipeline, which required extensive support from engineering and informatics.

For this project at a large academic cancer center, the challenge wasn’t just accuracy. It was ownership.

The team wanted a workflow where:

• A domain expert could define and refine the abstraction
• Iteration didn’t require writing code or working with an informatics expert
• AI outputs could be inspected, corrected, and improved directly by the person who understood the clinical nuance best.

‍

The Goal: Enable Independent, Expert-Led Iteration

‍

The objective was to reliably abstract “evidence of cancer” from oncology notes for a cohort of breast cancer patients while keeping the entire iteration loop in the hands of a non-technical researcher.

Rather than reducing the task to a binary classification, the researcher defined four clinically meaningful categories:

• Yes: documentation states or implies cancer is present
• No: documentation states or implies no evidence of cancer
• Uncertain: documentation is equivocal or indeterminate
• Not mentioned: cancer is not addressed in the note

These definitions mirrored existing human curation guidelines and reflected real-world clinical reasoning.

‍

The Brim Approach: AI Suggestions, Human Control

‍

Built and iterated by a domain expert

A non-technical end user configured the abstraction workflow in Brim without relying on informatics or engineering support.

Using Brim’s interface, she was able to:

• Define variable instructions and categories
• Generate AI-suggested labels
• Review supporting evidence directly in the source notes
• Identify where definitions were unclear or brittle
• Iteratively refine instructions and logic herself

No code. No prompt scripts. No handoffs.

‍

Iteration as a first-class workflow

Brim treated iteration not as cleanup, but as the core abstraction process:

• Evidence snippets made it clear why the AI proposed a given label
• Disagreements surfaced opportunities to clarify definitions
• Updates could be tested immediately on the same dataset

This allowed the researcher to improve results in real time, guided by clinical expertise.

‍

Dataset & Evaluation

‍

This project started by developing Brim variables by validating them against a golden dataset. This involved:

• 92 breast cancer patients
• Each patient had genetic testing and at least one year of follow-up
• 1,802 gold-standard curated labels
• Existing, detailed curation directives used as ground truth

This provided a rigorous benchmark for evaluating agreement, and for learning where abstraction logic needed refinement.

‍

Results: Strong Agreement and Iteration

‍

• Overall agreement: 91.5%
• Category-level agreement:
  • Yes: 96.4%
  • No: 85.8%

Key observations from review:

• Most discrepancies occurred in notes with ambiguous documentation
• Iterative refinement by the researcher improved alignment over time
• Second-pass review showed increased consistency once definitions were clarified

Importantly, improvements came from expert-led iteration, not model changes or engineering intervention.

‍

Brim is Built to Democratize AI-Guided Abstraction

‍

This case study highlights something subtle but critical:

The hardest part of chart abstraction isn’t running AI; it’s clarifying abstraction semantics.

Brim enables that clarification work to happen:

• Directly in the hands of domain experts
• Without requiring technical intermediaries
• With full transparency into AI behavior
• With fast feedback loops that reward careful thinking

Instead of locking logic into code or opaque prompts, Brim makes abstraction inspectable, debuggable, and improvable by the people who own the data.

‍

The Bigger Picture: Scaling Judgment, Not Just Automation

‍

Judgment-requiring variables like “evidence of cancer” appear everywhere: in registries, quality programs, and clinical research. They rarely fail because of insufficient compute. They instead fail because definitions are ambiguous, edge cases accumulate, and iteration is too slow.

This project shows a different model:

• AI generates candidates
• Humans iterate and refine
• Ownership stays with domain experts
• Rigor scales without engineering bottlenecks

That’s the kind of scale Brim enables.

‍

Interested in Expert-Led Chart Abstraction with Brim?

‍

If you're ready to empower your subject matter experts with AI-guided abstraction workflows, Brim is ready to help.

Learn more at brimanalytics.com.

‍