You Don’t Need Better Models

If you’re in charge of AI at a large company, you’ve probably been asked whether the latest AI model has been adopted yet, after someone has seen it top all the benchmarks. Similarly, while talking to CXOs, we commonly get asked how to keep up with all the new AI news. With new models releasing seemingly weekly, it’s almost impossible not to ask yourself whether the AI agents you build today will be outdated by the time they ship.

Thankfully, the answer is likely no.

Old models are enough. We live in a crazy world, and every couple of months a new model releases that breaks all the benchmarks and seems perfect on paper. As it turns out, a model 1 or even 2 generations older than the state of the art is already good enough to process a majority of the operational workflows you need done right now, across Finance, HR, Sales, Procurement, Operations, and any other function your firm might have.

In this article, we break down what actually happens when putting an agent in production for accounts payable, and why a new model doesn’t change that much. At the end, we also explore when it would make sense to switch to a new model.

What an agent does

Let’s assume you have a 40-person AP team processing 30k invoices/month. A background agent can handle the process from the invoice’s arrival in the inbox all the way until the invoice is booked in the ERP. This includes:

Reading the invoice PDF and extracting the necessary data

Finding the PO and goods receipt

Matching the 3 documents

Coding the invoice to the GL accounts and cost centers

Checking the vendor terms

Booking it directly (or raising an exception for a person / sending it for approval)

Which of these steps requires the best possible model in the world? Reading a PDF? Matching 3 documents? Coding to one of 400 GL codes based on past years? Sending an email to the vendor about a missing PO?

All of these are possible with models that were available 2 generations ago.

The frontier labs are working on progressing models to the point where they can solve coding tasks that would take a week, do research in mathematics, or drive your computer for a few hours to complete a task. We aren’t trying to discredit how impressive those tasks are, but when it comes to matching an invoice to a PO, the development curve is already flat.

For context, Varick works with large companies ($1B to $100B in revenue, 1k to 100k employees), helping them adopt AI, from general strategy, to process re-engineering, to building, deploying, and managing their agents.

Where misses come from

Let’s assume the 30k invoices come through for a month and you run the AI agent in the background to see how it actually compares to your AP team. Let’s also assume you end up with 1,800 cases (about 6%) where the agent does something differently.

You go through them with your AP leads and sort them into buckets based on why the two differ. It typically looks something like this:

700 cases: missing context. The context needed to solve the case was not available (e.g. the vendor agreed on a new way of invoicing via email 2 years ago, the PO was changed but not in the ERP, the payment terms only exist in the head of one of the clerks).

500 cases: unmapped tasks. The task was not mapped in the process documentation (partial deliveries, credit memos, invoices that need to be split across legal entities, etc.). Since the agent is built from the process documentation, it can’t handle them.

350 cases: bad or missing data. The data in the systems was either incorrect or unavailable (GR not done yet, vendor master has 2 entries for the same vendor).

150 cases: the agent was right. Your team was wrong.

100 cases: the model was wrong. It was a handwritten invoice, it didn’t sum 60 line items correctly, etc.

In this hypothetical of 30,000 invoices a month, maybe 100 of the misses are because of the model. That’s 0.3% of invoices, and maybe half of them are fixable by upgrading to the newest flagship model.

The remaining 1,550 misses are fixable by understanding the process and engineering in the extra context (pulling from inboxes, people’s heads, exception handling, and the vendor master). All of that work is the same regardless of how good the model is.

We see the same pattern in other workflows, like deal desk quotes and onboarding packets. The model pile usually ends up being the smallest.

This is what diminishing marginal returns look like inside a workflow. Going from no agent to an agent with a “decent” model gets you from manual to 94% agreement. Going from a “decent” model to the newest model gets you maybe 50 more invoices per month.

How much a swap costs

We’ve often heard: “Put the newest model on anyway, it can’t hurt.” Unfortunately, it often can, for 2 distinct reasons.

**1. Tokens**

Assume it takes 40k tokens across all steps to process an invoice, with 30k invoices per month. That’s 1.2 billion tokens per month.

Let’s say the “decent” model costs $2 per million tokens and the newest flagship costs $10 (these are estimates as prices change, but a ~5x differential is usually the norm). That’s:

$2,400 vs. $12,000 per month

8 cents vs. 40 cents per invoice

If it takes a person $11 to process an invoice, the cost of either model for this one workflow is pretty trivial, but it stacks up quickly. If you’re doing this for 50 different workflows in your company, it’s $1.4M vs. $7.2M per year. That’s an extra $5.8M per year for a marginal increase in outcomes.

**2. Retesting**

An agent is made up of the model + prompts + tools + eval-suite + sign-off from the controller that it’s good to go. Whenever you change the model, you have to re-test all of that.

Why? Because every model is a little bit different. Sometimes they output things differently. Sometimes they use tools differently. Sometimes they’re chatty, sometimes they’re not. Sometimes they’re unwilling to do certain things. Sometimes they’re worse at your specific task (yes, model regressions are real).

None of that is free. You need an eval-suite, you may need to fix some prompts, and finally you need to run it against your team again to see how it performs.

Across the big labs, it’s not unreasonable to expect a new flagship or “point release” every quarter. If you try to keep up with every update, that’s 4 adoptions per year, resulting in 8 weeks of redoing all of this for each workflow. That’s 8 weeks of not working on the other 1,550 misses that had nothing to do with the model.

When should you upgrade?

Of course there are scenarios where you should upgrade, but they should happen on your own time, for your own reasons. Consider the following:

Pin your models. Know which model you’re using for each workflow. Don’t let your agents silently change to new models.

Create your own eval-suite. Pull 500–1,000 invoices out of your ERP, exceptions included, each with the answer the team signed off on. Don’t use public evals. They’re designed to measure the labs’ tasks, and yours are different.

Make your agents model agnostic. Have a layer between your agents and the model provider. Don’t use each vendor’s SDK directly.

Test new models on your eval-suite, and upgrade if: (a) it solves some of the “model pile” misses you were stuck on, or (b) it scores the same as your pinned model for less (most commonly the new “small” model).

Run new models in parallel on live volume before they post anything.

Keep a list of parked workflows. Track the workflows you parked because no model was good enough, like reconciling a 200-page master agreement and several years of amendments to the latest version. Rerun them when there’s a major new release. This is where frontier models may shine.

Old models do get switched off, but the labs generally publish a retirement schedule, usually with several months of notice. If you’ve pinned a model version and have your own eval-suite and a model-agnostic layer, switching models is a planned 1–2 week project, not a rebuild. Without them, a single deprecation notice could turn your finished agent into a migration project.

What to keep up with

While most people keep up with AI news, we would encourage you to look inward. New releases outside the company are more often than not noise. What matters are the workflows inside your company, where the work actually gets done.

Unfortunately, no one can track that except you. The approval chain that picked up a 3rd review after one mistake years ago, the volume that’s outrunning the team, the exceptions that showed up this quarter: this is the context that compounds in your agents.

An agent that knows how work gets done at your company (even one running on last year’s model) will outperform a frontier agent that doesn’t, on every single invoice.

Do you need to keep up with every model launch? Probably not. If you read a newsletter (we recommend ours) and spend 20 minutes a week on X, that should be enough. Someone needs to be at the frontier testing each new model on each new workflow, but it shouldn’t be you.

Why everyone is focused on the newest model

The labs are largely incentivized to sell each new model with aggressive marketing at launch. There’s no better way to show how effective a new model is than with a graph of the tasks where it’s still improving. That’s fair game at the frontier, but nobody at a launch event is going to tell you that your quote-to-cash process was solved 2 releases ago.

Point-solution vendors arguably need the launches even more. If a vendor doesn’t train its own models, its only launch news is re-releasing the same product “powered by” the new frontier model.

Inside a company, “wait until the next model comes out, then we can do it out of the box” is the easiest way to put off building something. When a new model comes out, everyone needs to test it on their own workflow, and nothing ever gets deployed.

People often worry about competitors being on the newest model before they are, but the real competitors are the ones with 20 workflows in production on last year’s model, while everyone else is still comparing benchmarks and evals.

Ready to take the leap and ship your own agents? This is exactly what we do at Varick. We work with companies doing over $1B in revenue, across finance, sales, procurement, operations, HR, and more. 100% of our clients have gone to production, and 100% are now working on project #2.

Check us out at varickagents.com, and for more content like this, subscribe to our newsletter at varickagents.com/newsletter.

TL;DR

Use older models. Nearly every operational workflow can run on models 1–2 versions behind. You don’t need the latest and greatest to read a PDF, match 3 documents, and pick the correct GL code.

Sort failures by cause. Take a month’s worth of missed items from an agent and group them by failure. In the AP example, of 1,800 disagreements on 30,000 invoices, only 100 (0.3%) were the model’s fault. The rest came from missing context, unmapped exceptions, or bad data in the system of record.

The newest model costs you in 2 ways. Token costs ($0.40 vs. $0.08 per invoice seems small until you roll it out across 50 workflows, which adds up to $5.8M a year) and retesting (running your eval-suite, fixing prompts, and running in shadow mode for 2 weeks).

Pin, eval, and abstract. Pin model versions, build an eval-suite (500–1,000 cases), and keep agents model agnostic. Only upgrade when it tackles your model pile, delivers the same score for less, or your pinned model is being retired.

Keep up with your processes; no one can do that for you. For models, subscribe to a newsletter and spend 20 minutes a week on X.

Key Takeaways

_Imported from X — add takeaways on review._