Cutting an AI pipeline’s cost 4x in Rust, without flying blind on precision

Three weeks ago I sat down to audit GTM Intel's LLM bill, expecting to find some obvious waste and shave off a few dollars. What I found instead was that our cheapest model was quietly costing us judgment we hadn’t priced in, and that the actual fix touched several layers of a Rust pipeline that scans social platforms for buyer intent all day. This is the writeup of that audit: what we built to filter cheaply before any paid call, and the one tradeoff we shipped anyway, with a code comment attached admitting we haven’t fully closed the loop on it.

Where the money actually goes

GTM Intel watches Reddit, Bluesky, Hacker News, and X for posts that look like a buyer asking for help, then scores and routes them through a review queue. The cost breakdown was almost entirely LLM calls, and almost entirely one avoidable pattern: the system prompt on our classify call runs 700 to 1,100 tokens, and it gets sent fresh on every single post, even though most posts are 50 to 100 tokens of actual content. We were paying for the instructions over and over, tens of thousands of times, to evaluate things that were often obviously not a match.

The per-call price spread between models we route through OpenRouter is not subtle:

Roughly a 30x spread between the most and least expensive option, for calls that, on the surface, do the same shaped job: read a short post, decide if it matters. The obvious move is “route everything through the cheapest model.” The obvious move is also how you quietly ship worse leads to paying customers, which is the part that took longer to get right than the routing itself.

Filtering before any model sees the post

The biggest lever wasn’t a cheaper model, it was not calling a model at all for the 70 to 80 percent of posts that are obviously not buyer intent. We added a local embedding pre-filter that runs in-process, no network call, no per-token cost:

fn init_embedder() -> Result<Mutex<Embedder>, String> {
    let mut model = TextEmbedding::try_new(
        TextInitOptions::new(EmbeddingModel::AllMiniLML6V2)
            .with_cache_dir(cache_dir)
            .with_show_download_progress(false),
    )?;
    let exemplar_embeddings = model.embed(&exemplars, None)?;
    Ok(Mutex::new(Embedder { model, exemplar_embeddings }))
}

pub fn cosine_similarity(left: &[f32], right: &[f32]) -> f64 {
    let (dot, left_norm, right_norm) = left.iter().zip(right).fold(
        (0.0_f64, 0.0_f64, 0.0_f64),
        |(dot, ln, rn), (&l, &r)| {
            let (l, r) = (f64::from(l), f64::from(r));
            (dot + l * r, ln + l * l, rn + r * r)
        },
    );
    if left_norm == 0.0 || right_norm == 0.0 { return 0.0; }
    dot / (left_norm.sqrt() * right_norm.sqrt())
}

It’s a small ONNX model (fastembed’s AllMiniLML6V2) running fully in-process on an 8GB box, scored against 25 hand-written example buyer-intent posts (“our pipeline is dry this quarter, what tools work?”, that kind of thing), extendable per market via a JSON file. A post’s score is its best cosine similarity against any exemplar. Below a threshold, it never reaches a paid call.

We shipped it in shadow mode first, not enforce mode. In shadow mode the filter scores every post and writes a decision log, would_skip: true or false, alongside what the real triage pipeline actually did, but it never blocks anything:

pub struct DecisionLog<'a> {
    pub source: &'a str,
    pub sub_source: &'a str,
    score: LoggedScore,
    pub threshold: f64,
    pub would_skip: bool,
    pub triage_admitted: bool,
    pub post_chars: usize,
}

The point of shadow mode is boring on purpose: it lets us compare “the filter would have rejected this” against “the real pipeline actually admitted this as a lead” on real production traffic, before the filter has the power to silently drop a good post. Once that log has enough volume to trust the recall number, flipping LEAD_EMBED_FILTER_MODE from shadow to enforce is a one-line env change. Skipping this step and going straight to enforce would have been faster and is exactly the kind of shortcut that turns into a support ticket three weeks later from a customer whose best lead never showed up.

Splitting triage from classify, and the tradeoff we shipped with a warning label

Not every LLM call is equally sensitive to being wrong. A community scan might triage a few thousand short posts a day in a cheap first pass, deciding “does this even smell relevant”, before a smaller admitted set goes through a slower classify step that decides “is this actually worth a human’s review queue.” We split those into two independently configurable models:

model_classify: std::env::var("LEAD_MODEL_CLASSIFY")
    .unwrap_or_else(|_| "z-ai/glm-5.2".to_string()),
model_triage: std::env::var("LEAD_MODEL_TRIAGE")
    .unwrap_or_else(|_| "deepseek/deepseek-v4-flash".to_string()),

DeepSeek for the high-volume triage pass was an easy call: a same-corpus canary run showed identical accept/reject decisions against our GLM control on ten frozen Reddit posts, at roughly 79% lower cost per call. When two models agree on the same data, take the cheap one.

Classify is where I have to be honest, because the code itself is honest about it. Two and a half weeks earlier, a kaizen run replayed 502 frozen posts through the real pipeline with only the classify model swapped, deepseek vs sonnet, everything else identical. Deepseek’s two worst leads were also its two highest-scored: a business coach at 90 and a dental clinic at 92, both wrong-buyer false positives sitting at the top of what would have been a paying customer’s results page. Sonnet rejected both outright, with legible reasoning attached to the rejection, and judged precision on that run went from roughly 50% to roughly 100%. Cost ratio between the two was about 22x.

We still shipped GLM 5.2, not Sonnet, as the classify default. It’s cheaper than Sonnet, and it hadn’t been the model in that specific comparison, so we don’t actually know yet where it lands on the same precision test. That gap is sitting in the code as a comment, not swept under anything:

// Cost call: GLM 5.2 replaces sonnet here to cut per-post spend.
// The 20260702-123143 audit-kaizen run measured sonnet as best on
// wrong-buyer false positives, so re-run the kaizen harness against
// this model before trusting precision-sensitive changes.
model_classify: std::env::var("LEAD_MODEL_CLASSIFY")
    .unwrap_or_else(|_| "z-ai/glm-5.2".to_string()),

I’d rather ship that comment than pretend the tradeoff is settled. The rule we’re holding ourselves to going forward: no model swap on a precision-sensitive call ships as a quiet default change again without a kaizen replay attached to the commit, in either direction.

An insurance policy, not an optimization

None of the above stops a bad day, a misconfigured scan, a runaway loop, from spending real money before anyone notices. So the cost log doubles as a circuit breaker:

pub fn daily_budget_exceeded(cap_usd: f64) -> bool {
    cap_usd > 0.0 && spend_today_usd() >= cap_usd
}

spend_today_usd() reads the same append-only JSONL log, sums today’s USD field, and fails open (returns false, never blocks) if the log itself is unreadable, because a broken cost log should never be the reason a customer’s scan silently stops running. The worker checks this before starting new scan work, and the agent tool layer checks it before running a scan on demand, both refusing new spend past the configured cap until the next day. Cheap to build, and the only piece of this whole project that’s pure downside protection rather than a cost cut.

The numbers, and what’s still open

Still open, in roughly the order we’ll get to it: native DeepSeek API access for prompt-prefix caching, since we’re currently re-sending that 700-to-1,100-token system prompt through OpenRouter on every call instead of paying once for it; cross-tenant verdict caching, since “is this post a real buyer signal” is often account-agnostic and shouldn’t be re-judged per tenant; and Reddit access from our own datacenter IP, which Reddit currently blocks outright with TLS fingerprinting on top of the IP block, so unauthenticated JSON access needs either an approved OAuth app or a residential egress path, not just better headers.

The honest summary: the cheap model saves real money on the calls where being wrong costs nothing. On the call where being wrong means a paying customer’s top lead is a business coach instead of a buyer, we know exactly how much precision we’re trading for that savings, because we measured it, wrote it down, and left the number where the next engineer touching this file will trip over it.