AI coding agents are billed by usage, a bit like a meter running, and on the best models that meter runs fast. In our own setup, a one-line “say hello” to Claude Code on the top-tier model cost about 55 cents before any real work began, simply because the AI has to read its own instructions before it starts.

So the obvious idea is appealing: put a small, cheap AI decision model, like TypeSafe AI’s Jev, in front of the expensive one. Let it decide the code the model sees (retrieval), steer which tools it reaches for, and pick which model runs the session. Less waste, same results.

We built exactly that, then tested it properly against plain, unmodified Claude Code. Two of our three ideas made things more expensive. The third worked, but a much simpler rule beat it.

That is not the result we wanted. It is worth writing up precisely because of that.

The short version

• Giving the AI a “smarter” way to find information made every job more expensive, by 17% to 57% depending on the model and setup, with no improvement in success rate.

• The AI sorting the search results was not the problem. The search itself was. Switching the ranker off made no measurable difference. In more than half of searches, the right file was never in the results to rank.

• Handing work down to cheaper AI models is safe. Across 63 routed sessions, not one failed. On our first test set it cut cost by 57%.

• But “always use the mid-tier model” beat our clever router. On this task set, the mid-tier model never failed, so a fixed rule saved more (about 60%) than intelligent routing did (21% measured, about 43% once a test bug is accounted for).
  
  
  

What we built

A lightweight layer that sits between Claude Code and the AI models doing the work. It gave a small decision model three places, or “seats”, where it could act:

1. Retrieval: choosing which code snippets the main model sees, using a set of custom search and file-reading tools.

2. Steering: nudging the agent away from its built-in search and read tools and towards ours.

3. Model routing: deciding, at the start of each task, which model tier should do the work: Haiku 4.5 (cheapest), Sonnet 5 (mid-tier) or Fable 5.1 (most capable and most expensive). Think of them as a junior, an experienced professional and a senior specialist.

The decision model was Kev-4B, a small, free, open-source AI model. We originally designed the project around TypeSafe AI’s commercial Jev model, and used Kev as its open-source counterpart because it serves the same interface. To be clear: we tested Kev, not Jev. Kev’s own published accuracy sits slightly below Jev’s, so nothing here should be read as a verdict on Jev itself. We ran Kev on a single rented GPU.

Around it we built the plumbing needed to measure honestly: a meter on every request to the AI (how much it read and wrote, what it cost, how long it took), a record of every action the AI took, and a test bench to run it all. As a sanity check, the gateway’s cost figures matched Claude Code’s own reported cost to the cent on every run.

In short: a full TypeScript control plane (metering gateway on the Messages API, trace hooks into SQLite, an MCP retrieval server, a steering layer, a per-session model router) plus a benchmark harness running headless claude -p against 20 real bug-fix tasks from two open-source repos (openclaw, openai-agents-python), pinned commits, hidden-test verification, repeats, across Haiku 4.5 / Sonnet 5 / Fable 5.1. Two weeks, roughly $110 in cash (about $160 including subscription runs at list price).

How we tested it

We wanted real work, not toy prompts. We took 20 real bug fixes from merged pull requests in two active open-source projects, OpenClaw (TypeScript) and the OpenAI Agents SDK (Python). For each one we undid the fix, kept the project’s own automated checks, and wrote the task up as a bug report. A task only counts as solved if those checks pass afterwards.

The headline runs used “no-hints” prompts, which do not name the file or function to fix. That forces the agent to go looking, which is exactly where smarter retrieval should help.

Every task ran multiple times in each configuration, because the same task can pass on one attempt and fail on the next. One run proves very little. We compare rates over repeats.

Finding 1: Smarter retrieval made everything more expensive

This was the idea we were most confident in, and it failed most clearly.

With our retrieval and steering tools switched on, success rates stayed essentially the same, but cost went up every time:

The cause was not the amount of code being read. It was the number of steps. The model used our tools on top of its built-in ones, not instead of them. When steering blocked a built-in tool, the model spent another step retrying a different way. And with every extra step, the AI re-reads the whole conversation so far, so steps are where the cost lives.

Forcing the issue made it worse. When we blocked the built-in tools entirely, every habitual action became a denial followed by a retry.

The uncomfortable conclusion: Claude Code’s built-in tools are already efficient readers on this kind of work. Adding a layer on top, however clever, added steps and therefore cost. It is a bit like giving a capable employee an assistant they didn’t ask for: you add a meeting, you don’t save one.

Finding 2: The ranker was never the bottleneck. The question was.

If the extra layer did not help, perhaps the part that sorted the search results simply was not good enough? We tested that two ways.

First, we switched Kev’s ranking off and re-ran the same tasks. Cost and pass rate were the same, within normal run-to-run noise.

Second, we replayed 38 real searches the agent had made, comparing Kev’s ranking with plain keyword ranking. Kev put the right file first 39% of the time versus 34% for keywords, and was rarely worse. A modest edge.

But the bigger finding was underneath: in roughly 55% of searches, neither ranking contained the right file at all. The agent was searching for the wrong thing. No ranker, however good, can rescue a search for the wrong thing.

If retrieval is worth revisiting, it should be rebuilt around helping the model ask better questions, not around ranking the answers.

Finding 3: Routing down to cheaper models is safe, and it pays

The third seat was different. Here the decision model picks which model runs the whole task, once, at the start. Routing at that point (rather than mid-task) matters, because switching models mid-conversation throws away the AI’s short-term memory of the work so far, which is what keeps long jobs affordable.

We asked for the top-tier model every time and let Kev decide whether to route down.

• First test (6 tasks): routed sessions passed 24 out of 24, the same 100% as always using Fable, at 57% lower cost.

• Full test (all 20 tasks): Kev moved 67% of sessions off Fable onto Sonnet, and every completed session passed.

Across both studies: 63 routed sessions, zero failures. On ordinary bug-fix work, routing down is safe, and the saving is real.

Finding 4: Sonnet 5 was the cheapest model that never failed

To understand which model each task actually needed, we ran every task on every tier without any routing.

Haiku stumbled at least once on 8 of the 20 tasks. Sonnet and Fable never missed. That makes Sonnet the cheapest model that solved everything, and “always use Sonnet” the benchmark any router has to beat.

Here is how the savings compare against always using Fable:

• About 43% saved, estimated. The same routing decisions, priced at what each task normally costs on the chosen model.

• About 60% saved, ceiling. Simply always using Sonnet.

The router could not beat the simple rule on this set, because there was nothing for it to protect against. A router’s real value is insurance: catching the hard tasks that only the top model can solve.

What this means if you are buying or building AI tooling

Measure before you optimise. We built metering and a fair benchmark before writing a single optimisation. Without that baseline, our retrieval layer would have looked clever and cost money.

Count steps, not just tokens. In agentic systems, every extra round trip re-reads the conversation. A tool that saves bytes but adds a step is usually a loss.

Always test the boring option. “Always use the mid-tier model” took one line of configuration and beat a purpose-built routing system on this workload.

Price a router as insurance. If you deploy intelligent routing, know which tasks genuinely need your most expensive model, and test on them. If you have none, a fixed rule may be all you need.

Do not assume the smart component is the weak link. Our ranking model was fine. The failure was upstream, in how the agent formed its questions.

What comes next

Always use a mid-tier model. Sonnet passed 40 of 40, costs about 40% of what Fable does, and saved more than our own router (about 60% against Fable, versus our router’s 43%). Ship that tomorrow and you’ve already banked the win.

That changes the TypeSafe model’s job entirely. It stops being the thing that picks a model for every session. It becomes the thing that spots the rare monster and sends it up to Opus, then Fable. One question instead of twenty.

Two ways to build that escalation

Option A: decide before spending (predict)

A TypeSafe AI model looks at the task before a token is spent and answers one question: “will Sonnet fail this?” If yes, start on Opus; if clearly brutal, Fable.

Three things make it work, and the third is the one that matters:

1. Sharpen the question. We asked “how hard is this?”, unanswerable from prose. Ask only whether Sonnet will fail. Yes or no.

2. Give it the site survey. Jev currently sees only the ticket text. Feed it cheap repo signals too: files matching the symptom, subsystem size, test coverage, code churn. A builder quotes by looking at the house, not the email.

3. Calibrate the model on real outcomes. We have 155 labelled sessions sitting in our DB with pass/fail per tier, and Kev was never fitted to any of it. That’s why its confidence ran backwards: least sure on tasks Haiku aced, most sure where Haiku fumbled. We’ve been asking a bloke to guess form without ever showing him the race results. Fit the model to those outcomes and it stops guessing and starts knowing.

Cheaper in theory, no wasted attempt. But fragile: one needless escalation per 20 tasks wipes out the saving. Needs ~70% accuracy to beat Option B.

Option B: decide while spending (react)

Every session starts on Sonnet. Jev/Kev watches it work and promotes mid-flight only when it sees the model drowning: verify failing repeatedly, same file edited over and over, tool errors climbing, turn cap closing in. The doorman steps in when glasses start breaking.

Switching models mid-flight does mean the new model starts without the old one’s short-term memory of the work, but when we tested it, that didn’t make the run more expensive.

• Robust: no guesswork, it reacts to what actually happened.

• Costs ~5%: the wasted Sonnet attempt, about $0.38 per rescue. Over 20 tasks with one rescue, a perfect Option A costs ~$8.15 and Option B ~$8.53; that one wasted attempt is about 5% of the whole bill. More rescues, bigger the waste.

• Needs plumbing: routing is one-shot per session today; this needs per-request re-evaluation with hook signals wired in.

So the order is: ship Sonnet-always → hunt for the breaking task → then build A (calibrated) or B.

Where JustifyAI stands

Most conversations about AI cost focus on picking the right tool. This project is a reminder that the bigger wins, and the bigger traps, sit in the architecture around the model: what gets measured, where decisions are made, and which “obvious” optimisations actually hold up under a fair test.

That is the work we help organisations do: separating real AI savings from plausible-sounding ones, and designing systems that are measured honestly from day one.

If you are trying to bring AI costs under control without hurting results, get in touch with JustifyAI or subscribe to The JustifyAI Brief for more analysis like this.

Notes on method

• All dollar figures are at Anthropic list prices. Subscription runs are shown at their list-price equivalent.

• Pass rates exclude a small number of sessions that did not finish because of network outages, not model errors.

• Results come from 20 bug-fix tasks in two repositories. They describe this class of work and should not be generalised to every kind of coding task.

• The decision model tested was the open-source Kev-4B, not TypeSafe AI’s Jev.

• Benchmark repositories: OpenClaw and OpenAI Agents SDK for Python. Decision model: Kev