← sources

Most production “AI agents” are classifiers with extra steps, and a classifier can give you guarantees

source: reddit (reddit:datascience)published 2026-10-12 07:47ingested 2026-10-13 02:37

retained text

Routing a ticket or a claim to the right queue is a classification problem. Using an autoregressive LLM for it means paying for decoding, parsing JSON that sometimes breaks, and getting no real statement about how often it is right. A small encoder with conformal prediction gives you prediction sets with a coverage rate you can actually state, such as 95%, and you only escalate to an LLM when the set is ambiguous.

Found a workshop on Oct 30 that builds exactly this on CPU as one case study.

Covers:

* Why generative routing is an engineering anti pattern, and how to separate vector scoring from state execution
* Zero shot and few shot routing with ModernBERT, LFM2.5 Encoder and SetFit
* Split conformal prediction to turn raw logits into sets with guaranteed coverage
* Idempotent LangGraph execution with human approval gates and compensation handlers
* Evaluation with Wilson score intervals and clustered standard errors instead of a naive pass rate

Led by Ben Auffarth, who has 15+ years across machine learning and large scale decision systems. It sits at an intermediate level, so some Python and transformer familiarity is assumed. 

Full details and the agenda are in comments

document fetched 2026-10-13 00:24 · canonical https://reddit.com/r/datascience/comments/1x3vs98/most_production_ai_agents_are_classifiers_with