Case Study: Cutting LLM Costs by 90% Without Losing Quality

I run Kuverly, a social listening tool that ingests huge volumes of posts and surfaces only the relevant few to customers. That relevance judgment ran through a multi-shot LLM pipeline — several passes per post, tuned hard for quality.

This is the process I used to cut our own LLM bill by 90% with zero quality loss — the same process I now apply for clients facing the same problem.

The Problem

I was iterating on prompt quality, not cost. Multi-shot prompting kept improving relevance, so I kept using it. Nobody was watching the meter.

Then I looked at the daily bill: $2 per customer per day in LLM costs — more than the plan itself brought in per customer, per day. Unsustainable, and urgent.

The Fix

Cutting quality wasn't an option — the expensive pipeline was expensive because it worked. The fix had to filter volume without touching judgment on anything ambiguous.

This is the same process I use with clients, in priority order: guard, embed, cascade. Each targets a different kind of waste in an LLM pipeline, but they aren't equal — some produce bigger, more durable savings than others.

1. Guard — stop unnecessary calls before they reach the LLM

A lot of LLM problems, when you look closely, are actually problems traditional ML already solves well — not just yes/no calls, but multi-class decisions, scoring, ranking, anything a classifier or model can be trained to do reliably. Where that's true, you don't need the LLM to make the call every time; you need a guard in front of it.

I built a classifier that sits in front of the LLM flow with one job: reject, never accept.

  • Trained on the LLM's own past decisions, plus higher-weighted signals from user feedback
  • Set to a high confidence threshold
  • If it rejects, the post is dropped — no LLM call
  • If it's not confident, the post still goes through the full LLM flow, unchanged

The bar for the classifier was never "beat the LLM" — just match it on the easy cases. Matching means most of the volume gets filtered for pennies, and the LLM is reserved for posts that actually need it. User signals were added to give the classifier a shot at outperforming the LLM too, but that was a bonus, not the requirement.

Under the hood, it runs as a king-of-the-hill setup: separate classifiers retrain on new data and compete against the live model, and the best performer stays in production.

The guard did most of the heavy lifting — shipped in 2 days, since the training data already existed from the LLM's own history.

2. Embed — skip the LLM entirely where similarity is enough

Some questions never needed reasoning, just similarity. Topic matching and article suggestion moved from LLM calls to embedding distance — a fraction of the cost, same result.

Embeddings are the most underused lever. They're universally cheaper than an LLM call, and one embedding can answer several different questions — the same vector that powers topic matching can also power article suggestion, deduplication, or search, without paying for a new LLM call each time.

3. Cascade — stop making your biggest model answer every question

Instead of one large model handling every question in a big multi-shot prompt, I split the work into more, narrower questions — and routed each to the cheapest model capable of answering it. The large model was reserved for what actually needed it; smaller, cheaper models picked up the rest.

Cascading is the easiest win to reach for — often an immediate ~50% saving just from using the cheapest model that can still do the job. But it's the least durable of the three: it carries real quality risk if pushed too far, and it isn't immune to the market — cheap models get deprecated or replaced, and today's saving can quietly erode.

The Result

  • 99% match rate between what the guard rejects and what the LLM would have rejected anyway
  • Daily cost per customer: $2 → $0.20 (90% reduction)
  • No drop in post quality — the LLM still makes every judgment call that matters

Why This Applies Beyond Kuverly

None of these three levers are specific to social listening — they apply to any product where an LLM is making high-volume decisions:

  • Guard works anywhere an LLM is repeatedly making a decision that traditional ML can also make — not just binary yes/no, but multi-class categorization, scoring, or ranking at high volume. A trained model can absorb most of that load for a fraction of the cost.
  • Embed works anywhere the task is really about similarity or matching, not reasoning — and pays off repeatedly, since the same embeddings can answer more than one question.
  • Cascade works anywhere a single large model is answering questions that don't all need its full capability — fast to implement, but the savings need monitoring as models change.

Most LLM-heavy products are paying premium-model prices for a lot of work that doesn't need a premium model. The fix isn't "use AI less" — it's routing each piece of work to the cheapest process that can still do it right.

If your LLM costs are eating your margin, this is exactly the kind of audit I do for clients: find where the expensive model is doing unnecessary work, and restructure the pipeline so the expensive model is used only where it's actually earning its cost.