Why Manual and NLP-Based Chart Abstraction are Becoming Obsolete

December 11, 2024

Natural Language Processing is our Current Alternative to Manual Abstraction
Getting structured data is often the first step to discovering new medical knowledge and improving patient care. Historically, this chart abstraction process has been done in one of two ways:
- Manually. Hire trained personnel to comb through medical records for information and enter it into a database or spreadsheet.
- Natural Language Processing (NLP). Build a rules-based model using NLP to extract and code the data.
At Brim Analytics, we’re leading the transition to guiding chart abstraction with Large Language Models (LLMs), which has the potential to outperform both manual and NLP-based chart abstraction.
We usually focus on the advantages of LLM-guided abstraction over manual methods, but experts in the field also frequently ask why LLM-guided abstraction is superior to NLP. Let's dive in.
LLMs Are More Generalizable, Contextual, and Iterative
Traditional NLP systems rely heavily on predefined rules, or models that require extensive customization for each use case or document type. These systems struggle with context, particularly for complex or ambiguous medical statements, often requiring manual feature engineering or additional specialized models.
In contrast, LLM-guided chart abstraction tools excel for a few reasons.
Generalizability. The same tool can handle chart abstraction across multiple domains effectively without requiring lengthy domain-specific training.
Contextual Understanding. LLMs interpret varied wordings and phrasings with ease, much like a human reviewer, whereas NLP systems often stumble without explicit modifications.
Quick Iteration and Improvement. Tools can leverage labeled examples using in-context learning to quickly improve if they make errors. In Brim, correcting errors in LLM judgments is as simple as providing feedback and regenerating outputs.
LLM-Guided Tools Can Be Deployed Quickly Without On-Site Technical Expertise
Deploying an NLP tool for a new use case involves gathering training data, customizing the model, and meticulously checking for edge cases. Post-launch modifications demand expertise in NLP, which can introduce delays and costs.
In contrast, tools like Brim’s LLM-guided solutions can be deployed rapidly without requiring specialized model training. The chart abstraction rules adapt seamlessly to feedback, reducing the need for ongoing technical support.
LLMs Enable Continuous Improvement and Multi-Modal Opportunities
The current wave of investment in generalized LLM development (eg Claude, ChatGPT, Nora and others) has the potential for continued improvement in their capabilities. Epoch AI estimates that training compute grew 4-5x/year for this kind of model over the past 5+ years. Leveraging these advancements means chart abstraction tools built on LLMs could see ongoing gains in accuracy with minimal effort.
Additionally, many LLMs integrate with multi-modal inputs and outputs, enabling new abstraction possibilities. For example, these tools can analyze images alongside text as part of a chart abstraction workflow, opening up future avenues for comprehensive data extraction.
Purpose-Built Tools Effectively Address Challenges with LLMs
There are a few common challenges when working with LLMs:
- Interpretability and Hallucinations. Generic LLM chat interfaces can make it hard to understand the reasoning behind an LLM's conclusions. Sometimes these conclusions appear to have no evidentiary basis. In many applications, this is not important, but chart abstraction requires accurate and traceable results.
- Cost. Training base models is very expensive, and initially Generative AI Model creators passed that cost on to customers.
The cost of using LLM endpoints has come down dramatically, with GPT-4o mini from OpenAI marking a 99% cost drop in 2 years for better performance. We believe that the cost barrier is significantly lower than it was a year ago, and that it will continue to go down as models get more efficient.
That leaves interpretability and hallucinations.
At Brim Analytics, we're building an LLM-guided solution rather than a specialized model so that we can built our tools with guardrails to support the accuracy of chart abstraction. Two of Brim's major guardrails are:
- Traceability. Every extracted data point references the specific raw text from the medical note that informed the decision, ensuring transparency.
- Human-in-the-Loop Design. Our tools empower chart reviewers to actively validate and provide feedback on data points, improving the model’s accuracy over time.
Brim will continue to feature architectural choices that limit the impact of LLM drawbacks while effectively leveraging the strengths and scalability of these models.
Conclusion
Large Language Models can drive a paradigm shift in chart abstraction, outclassing manual and NLP-based methods in speed, accuracy, and adaptability. Tools like Brim’s LLM-guided solutions bring generalizability, contextual understanding, and iterative feedback to the forefront, while addressing common challenges with innovative design. With LLMs, the future of medical data abstraction is faster, smarter, and more reliable than ever before.