Model Routing for Support Bots: Cheap-First FAQ Handling

extract

Model Routing for Support Bots: Cheap-First FAQ Handling OpenRouter · Cheap-first model routing sends routine support questions to a small, inexpensive model and escalates selected hard or uncertain requests to a stronger model. It can reduce the cost of using a capable model for every message, as long as the cheaper path meets your support requirements. The three application-side patterns are static rules, classifier-based triage, and answer checks with confidence thresholds. This guide compares the three patterns, shows which parts of the flow OpenRouter handles, walks through a support flow with a copyable request, and sets out what to measure before you add escalation. Tl;dr - Use static rules for narrow, recognizable FAQs, classifier-based triage for stable categories expressed in varied wording, and answer checks when a cheap model should attempt the response first. - Keep support-policy acceptance checks in your application. Our model fallbacks handle request errors. A successful but incorrect answer needs a separate escalation decision in your code. - Compare cheap-first against cheap-only and stronger-only on answer acceptance, total cost including discarded attempts, escalation rate, and full-turn latency. - Add escalation only where your evaluation shows it repairs failures. An escalation that replaces an already acceptable cheap answer adds cost and latency without improving the response. When a cheaper model fits routine FAQs One capable model keeps the initial integration simple. If routine FAQs meet your quality requirements on a cheaper model, that traffic becomes a candidate for lower model spend. Test both models against the same approved documentation, using candidates from our model catalog and current pricing. Classifier calls and second attempts also consume time and budget. Choose a pattern based on what you can identify before generation and what you can check afterward. Three routing patterns Anthropic’s guidance on building effective agents describes routing as classifying an input and directing it to a specialized follow-up task, with easy questions sent to smaller models and hard questions to more capable ones. The build and maintenance estimates below are our planning judgments for a small support application. These patterns can coexist. Rules or classification choose where a request starts. Answer checks decide whether its output is acceptable. A model’s self-reported confidence is not a probability of correctness until you have checked it against correctness on your own questions, so treat it as a ranking signal until you have that data. What your application owns and what OpenRouter handles Owning quality-based routing means maintaining rules or a classifier, acceptance checks, escalation decisions, and monitoring. These can live inside your support application without a separate routing service. If you want us to select the initial model, our Auto Router, selected with "model": "openrouter/auto", classifies each prompt by task type and picks a model from the ones OpenRouter users spend on for that task type. An ordered model fallback list, sent as the models array, retries the request on the next model after an error such as a rate limit, provider downtime, or a content moderation refusal. Support-policy acceptance checks remain your responsibility. Choose the Auto Router if you want us to select the initial model. Choose an ordered fallback list if you know your preferred models and need error recovery. For cheap-first quality escalation, your application checks the candidate and requests another answer when the check fails. The following flow shows that division of responsibility. A worked support flow on OpenRouter Step 1: Define the outcome before choosing a model HarborDesk is a fictional SaaS product with a fixed FAQ. Its bot can explain policies and recommend support, but it cannot inspect accounts or make changes. An acceptable turn answers the question, asks for clarification, or recommends human support. Use approved templates for intents you recognize reliably and that have fixed answers. Test intent selection separately so the correct wording reaches the correct question. Requests that need interpretation enter the generated-answer path below. Step 2: Check generated answers before escalating For requests that need generation, the cheap model provides the first candidate and the triage signals. Your application owns the acceptance decision. Consider a customer who asks how to cancel a subscription and request a refund. In HarborDesk’s policy, owners cancel under Settings > Billing, and billing staff review refund requests. - Accept a cheap answer that explains both steps without promising a refund. - Withhold an answer that sends everything to billing and omits owner cancellation. Use an approved template if one covers the request, or request a stronger-model attempt if your evaluation shows the stronger model fixes this omission. - Recommend billing support if the customer asks the bot to approve the refund. A stronger model cannot supply that authority. Check the policy content and the intended action. Valid JSON alone does not establish correctness. Preserve conversation context, run the same checks on the stronger answer, and stop after one escalation. Step 3: Configure error fallback and record request cost This snippet sends one request with an ordered fallback list. Install requests and set OPENROUTER_API_KEY before running it. The short fictional FAQ supplies the context for the example. import os import requests messages = [ { "role": "system", "content": ( "Answer only from this fictional HarborDesk FAQ. " "Owners can cancel under Settings > Billing. " "Billing support reviews refund requests. " "You cannot inspect accounts, cancel plans, or approve refunds. " "Ask for clarification when necessary." ), }, {"role": "user", "content": "How do I cancel my subscription and request a refund?"}, ] response = requests.post( "https://openrouter.ai/api/v1/chat/completions", headers={"Authorization": "Bearer " + os.environ["OPENROUTER_API_KEY"]}, json={ "models": ["openai/gpt-4.1-mini", "openai/gpt-4.1"], "messages": messages, "max_tokens": 512, "stream": False, }, timeout=60, ) response.raise_for_status() data = response.json() if "error" in data: raise RuntimeError(f"Completion failed: {data['error']}") choices = data.get("choices", []) if not choices: raise RuntimeError("Completion returned no choices") choice = choices[0] if choice.get("error") is not None or choice.get("finish_reason") == "error": raise RuntimeError(f"Completion failed: {choice.get('error', 'unknown error')}") print({ "id": data["id"], "model": data["model"], "answer": choice["message"]["content"], "cost": data.get("usage", {}).get("cost"), }) The models array tries the next model after an eligible error, such as a rate limit or provider downtime. A successful but incorrect answer does not trigger it. Quality escalation requires an application check and a separate stronger-model request. The two error checks exist because we can return HTTP 200 when the model started processing the request and failed while producing output. The error object appears at the top level of the body, or inside choices[0] with finish_reason set to error when partial content exists. The errors and debugging reference describes both shapes. Treat both as failures rather than printing partial output as an answer. The snippet prints an unvalidated candidate for inspection. The usage.cost field is the amount we charged for the request in credits, and model is the model that produced the response, which matters when the fallback fired. Log both for every request. When your application escalates, sum both calls, including the discarded cheap draft. If a response arrives without a cost value, record the cost as unknown rather than zero until you reconcile it against your activity page. The usage accounting cookbook describes the usage object. Measure escalation rate and cost per resolved ticket For a pipeline that always tries the cheap model first, use this formula for the expected model cost per request. expected_cost = cost_cheap + cost_check_1 + escalation_fraction * (cost_strong + cost_check_2) Include the cost of any evaluator or classifier calls in the check terms, and use the stronger model’s cost on escalated requests. Compare the result against cheap-only and stronger-only, including their checks, at the same quality target and latency limit. Track these measures for each routing policy. - Escalation rate. The share of requests that took a stronger-model attempt. - Incorrect answers accepted. Cheap answers that passed your checks but failed human review. - Correct answers escalated. Cheap answers that already met your criteria but were escalated anyway. Each one adds a stronger-model charge and a second call’s latency without improving the response. - Handoff appropriateness. Whether the bot recommended human support when the request needed authority it does not have. - Full-turn latency. Time from the customer’s message to the returned answer, including both model stages on escalated requests. A high escalation rate warrants inspecting the routed questions. It does not by itself prove the threshold is wrong. For cost per resolved ticket, divide all model spending for a ticket cohort, including unresolved tickets, by the number of tickets that met a stated resolution definition within a stated reopen window. Answer acceptance on a test set measures response quality, not customer resolution, so keep the two metrics separate. Run this comparison on held-out support questions before you enable escalation in production. Include straightforward FAQs, ambiguous requests, troubleshooting, and requests that need a human handoff, and grade each answer against predefined content criteria and the expected action. Escalation earns its place only where it converts failing answers into passing ones at a cost and latency you accept. Conclusion Begin with approved templates and a cheap-model baseline for a narrow FAQ. Choose rules, classification, or answer checks based on your workload, then compare against stronger models only at the same quality target and latency limit. Add escalation only where it fixes enough failures to justify the cost and delay. Revisit your checks when support policies change. Frequently asked questions Can I avoid building a custom routing service? You can keep routing inside your support application. Use the Auto Router if you want us to pick the initial model, and use the models fallback array so we retry another model after a request error. The support-specific acceptance checks, such as whether an answer follows your cancellation policy, still belong in your application. Keep model IDs in configuration and keep your prompts and evaluation data portable so you can change models later. Can a cheap model consult a stronger model mid-generation? Not through the flow in this guide. The cheap model finishes a candidate answer first, then your application decides whether to request a stronger-model answer. Consulting another model during a turn requires an explicit tool or orchestration step in your code that calls the stronger model and returns its output to the ongoing workflow. The models fallback array does not create that mechanism. It only retries the request on another model after an error. Which FAQ triage model should I choose? Start with the cheapest candidate that meets your quality target on held-out support questions, including ambiguous requests and policy exceptions. Measure full-turn latency rather than single-call latency, because the fastest model call can still produce a slower support response if it often needs a second attempt. Begin with a small set of recurring questions and expand as your evaluation shows which routes work.