Best Models for Brim: OpenAI Generation 4-6
October 15, 2026
Best Models for Brim: OpenAI Generation 4-6
“Which LLM model should we use with Brim?” is one of the questions we get most often from Brim customers. Brim enables this flexibility with its Bring Your Own LLM design, but choosing which model to use is more complicated than you might think.
Choosing a Large Language Model typically involves three things that pull against each other: cost, speed, and intelligence. To add complexity, the way that these things interact depends on the specific problem the LLM is solving: for example, some models are faster at simple tasks but slow down when confronted with more difficult ones, and some of the “smarter” models spend more time on simpler tasks than you might expect.
The landscape of model intelligence is also rapidly changing. Models across every provider have become genuinely capable, and for a lot of tasks the gap between a frontier model and a small one is narrower than it was a year ago. So the useful question has become: which kind of intelligence does this particular job need?
Chart abstraction does not ask an LLM model to be broadly brilliant, especially in the BAMM framework, which breaks a chart abstraction task into several smaller LLM calls. To perform chart abstraction successfully, a model needs to read a chart carefully, interpret that chart even when the language is hedged, unclear, or involves acronyms, apply a coding rule the same way every time, and avoid guessing when the evidence is thin. Models differ on these specific skills in ways a general benchmark score will not show you.
The speed, cost, and intelligence tradeoff also differs depending on factors like note length, note complexity, field type, and more. The right model for doing chart abstraction for an ongoing registry might be different from the right model for a retrospective research project or clinical trial prescreening.
To build a body of research around the best models for Brim-style chart abstraction, we are starting a series of explorations comparing different models. This is the first.
Coverage and Caveats
This post compares six OpenAI models against each other when used in the Brim chart abstraction framework. We started with a single provider because the operative question for most customers is how to choose between models from the provider they already have a relationship with. We intend to expand this work to include other providers in the coming months.
Four caveats worth stating up front.
- This analysis is focused on chart abstraction, not general model performance. Our analysis focuses on performance on Brim chart abstraction. Nothing here should be read as a claim about these models in general.
- We intentionally focused on mini and nano class models. Brim decomposes an abstraction project into small, tightly scoped queries rather than asking one model to read a whole patient chart and reason about everything at once. That design makes smaller models viable.
- The N is small. We are starting this series using two datasets and one run per dataset per model. We intend to expand the testing datasets over time but feel that even with a limited N, the results are directional and helpful.
- We’re reporting deltas to a baseline of GPT-4.1-mini. The datasets we’re using concentrate adversarial cases that would be rare in a real chart, so absolute scores would not tell you what to expect on your own data. We’ve made the decision to report everything below as a delta to the baseline of GPT-4.1-mini.
The Test Datasets
For this analysis, we used two benchmark datasets which cover two different groups of difficult cases.
- The Value dataset requires reading a field out of pathology, operative, and treatment text and coding it with stage, grade, counts, lab values, regimens, etc. This is mostly a matter of locating evidence and applying a rule to it.
- The Date Resolution dataset requires reading event dates out of narrative notes where the date is often implied, partial, relative, or contradicted somewhere else in the chart. This is mostly a matter of reasoning about evidence.
Both datasets use synthetic colorectal cancer charts built to stress specific failure modes rather than to resemble a typical patient mix. Each patient is a small case constructed around one difficulty, and patients are grouped into families that share it.
Methods and Overall Findings
We ran GPT-4.1-mini as the baseline and compared four alternatives against it:
Every number below represents the change from the baseline on the same work.
Definitions:
- Run cost represents the dollar cost of one complete pass over a benchmark set. We multiply the tokens consumed when processing a dataset by that model’s published API input and output prices as of September 2026. This ignores negotiated prices, as well as different prices for cache read, cache write, and reasoning.
- Run time is comparing the time spent processing data for each dataset for each model on our test Brim instance.
- Color codes:
- 🟢 better than the baseline
- 🔴 worse than the baseline
- ⚪ no material change
Value dataset, % change vs. baseline
All of the models spent more time on this task than the baseline, but only GPT-6-luna got higher agreement as a result.
Date Resolution dataset, % change vs. baseline
On the Date Resolution dataset, all of the models had lower agreement than baseline, except for GPT-5.4-nano. This may be because GPT-5.4nano had a medium reasoning level; this is reflected in its runtime being the only one that increased over the baseline for dates.
The models demonstrated different strengths and weaknesses relative to the baseline of GPT-4.1-mini:
- GPT-4o-mini was the cheapest model in the 4 and 5 generation, but this came at a cost to accuracy.
- GPT-5.4-nano was cheaper and slower than gpt-4.1-mini and gpt-4o-mini, with accuracy comparable to gpt-4o-mini.
- GPT-5.4-mini was the most expensive model. It roughly matched GPT-5.6-luna and GPT-6-luna on accuracy.
- GPT-5.6-luna showed the accuracy of GPT-5.4-mini at a much better price and a slightly better speed.
- GPT-6-luna improves on GPT-5.6-luna on some accuracy tasks, but not others. It is a good value for money.
Speed Deep Dive: Speed is Workload-dependent
Speed depended on the workload.
On the Value Dataset, no model ran faster than GPT-4.1-mini (580 seconds). GPT-6-luna and GPT-5.6-luna came within 5% of it, while GPT-5.4-mini took 26% longer.
On Date Resolution, the newer models pulled ahead: GPT-5.6-luna finished 31% faster than the baseline, GPT-5.4-mini 25% faster, and GPT-6-luna 21% faster.
GPT-5.4-nano was the slowest model on both workloads, taking 156% longer on dates and 145% longer on values. Each model ran once per workload, so differences under 5% are within normal run-to-run variation.
Cost Deep Dive: Driven by Pricing, not Token Count
Tokens and cost are shown side by side here because they do not track each other. The token columns are how much each model consumed on the same work; the cost columns are what that consumption cost in dollars at list price.
List prices vary by more than 7x across the six models, and actual run costs mostly follow that order.
GPT-6-luna is the cheapest by a wide margin. Its input price is a quarter of GPT-4.1-mini’s, and it cost 77% to 81% less to run on both workloads.
GPT-4o-mini and GPT-5.6-luna also cut costs by more than half.
GPT-5.4-mini is the most expensive model, costing 74% to 95% more than the baseline.
GPT-5.4-nano saved a lot of price: 53% on the Value Dataset, and 43% on Date Resolution. This was primarily lower per token pricing.
Cost assumptions. Published list prices per million tokens, current as of September 2026.
Accuracy Deep Dive: Small, but Smart
Models in OpenAI’s 4th, 5th, and 6th generations are able to achieve pretty good accuracy out of the box, even though we only evaluated mini and nano models here.
On the Value Dataset, agreement was close across most models. GPT-6-luna, GPT-5.6-luna and GPT-5.4-mini all landed within 3% of GPT-4.1-mini’s 90.4%. GPT-4o-mini and GPT-5.4-nano trailed.
Date Resolution separated the models more. GPT-4o-mini fell furthest, at 51.2%. GPT-6-luna’s 11% drop on dates comes mostly from fields it left empty, not from wrong answers. It skipped about a quarter of date fields, but when it did return a date, it was right 88% of the time, compared to 80% for GPT-4.1-mini.
Which Model Should you Choose for your Brim Project?
Using these datasets, the strongest value for time and money are GPT-5.6-luna and GPT-6-luna. GPT-6-luna is the cheapest, and GPT-5.6-luna was nearly as cheap and with slightly better accuracy.
If you are seeing lower accuracy than you need and your variables are tuned, you can try increasing the reasoning level. This will shift more of the tokens to reasoning, making the run more expensive and slower, but it may boost accuracy as well.
If you're trying to pick a model for a specific project, or have your own model benchmarking work to share, we'd love to hear from you. Reach out to support@brimanalytics.com.