NarrateAI: production-ready LLM quality assurance on Amazon Bedrock

extract

Artificial Intelligence NarrateAI: production-ready LLM quality assurance on Amazon Bedrock Executives need to make data-driven decisions during live business reviews, where accuracy and speed matter. A conversational agentic AI assistant can meet this need by answering data questions instantly. But the stakes are high: a wrong number or a slow response in front of leadership carries immediate professional consequences, and a capable large language model (LLM) alone can’t guarantee either. In production, that gap shows up as hallucinated metrics, API throttling, validation latency, and subjective language. Closing it requires production-ready quality assurance built into every step, from data retrieval to response delivery. This post details five techniques, implemented on Amazon Bedrock, that work together to deliver it: adaptive pipeline orchestration, cross-account multi-model failover, real-time streaming evaluation, composite evaluation framework, and data accuracy verification. Each addresses a distinct failure mode while operating as a coordinated system. This post is the second in our NarrateAI series. NarrateAI transforms business intelligence for over 4,000 AWS executive leaders through a two-layer architecture built on Amazon Bedrock AgentCore, a platform to build, connect, and optimize agents at scale, with any framework or model. The architecture comprises an Automated Narrative Generation Layer for batch processing and a Conversational AI Interface Layer for real-time interaction. Our previous post covered the business challenges, overall architecture, user experience, and enterprise deployment. This one goes deeper into the engineering, showing how the five techniques achieve approximately 99 percent numerical accuracy while streaming responses in real time. It’s written for engineers and architects building LLM applications who are familiar with LLM APIs and streaming responses. From business vision to technical implementation This post focuses exclusively on the advanced quality assurance mechanisms within the real-time layer. Consider an executive asking “Which regions aren’t meeting their targets and why?” The five techniques work together so that the response is numerically accurate and arrives in real time. It also maintains throughput under concurrent global usage and is professionally formatted for direct use in business reviews. Figure 1 shows how these five techniques form a layered dependency chain, where each layer feeds the one below it to produce a validated real-time response from raw queries. Adaptive Pipeline Orchestration routes queries by data volume so that most complete in a single fast pass while complex queries receive full parallel treatment. Cross-Account Multi-Model Failover expands available inference capacity across independent model-account quota spaces, reducing user-visible throttling. Real-Time Streaming Evaluation validates each paragraph the moment it’s produced, overlapping quality checks with generation. The Composite Evaluation Framework runs multiple independent evaluators in parallel against each paragraph. Data Accuracy Verification catches numerical hallucinations through a two-stage cascade. It starts with cheap exact matching and escalates to semantic verification only when needed. The following sections walk through each technique in the order it appears in the pipeline, from data retrieval through generation to quality assurance. Adaptive pipeline orchestration Enterprise knowledge documents contain extensive information and queries vary enormously in how much of it they need. A focused question like “What’s my team’s quarterly attainment?” might draw on a handful of sections (discrete chunks of enterprise knowledge documents retrieved). A comprehensive request like “Give me a full regional performance analysis” requires synthesis across hundreds. Users expect the same response time either way. This heterogeneity requires a routing strategy that handles the common case quickly without sacrificing thoroughness for complex scenarios. No single processing strategy works for all queries, because the right approach depends on how much data a query actually requires. Single-pass concatenation is fast and cheap, but it breaks down when aggregated sections exceed the model’s context window (typically 200K tokens), forcing truncation and quality loss. Fixed multi-pass batch processing avoids truncation, but it pays for multiple LLM invocations on every query, including the roughly 90 percent that don’t need them. Our approach routes each query based on its total aggregated section volume ∣D∣ (where D is the set of retrieved document sections): single pass when the data fits, batch processing only when necessary. The result is a three-phase pipeline that classifies each query once and then processes it along the cheapest path that preserves quality. Three-phase adaptive processing The pipeline processes every query in up to three phases: Mode-Aware Consolidation packs retrieved sections and picks a route, Bifurcated Analysis executes that route as either a single fast-path call or parallel normal-path batches, and Conditional Consolidation merges the parallel results when needed. The following table summarizes how the two paths move through these phases before we walk through each phase in detail. The first phase, Mode-Aware Consolidation, operates on retrieved document sections. It applies a greedy first-fit packing strategy to fill context windows up to the token limit while preserving section priority order and section boundaries. For smaller volumes (∣D∣≤θ, where θ is an empirically calibrated character-count threshold), all sections are concatenated into a single chunk, and this is called “fast path”. For large volumes (∣D∣>θ), sections are packed into optimal batches that respect document boundaries and this is called “normal path”. Threshold θ is set relative to the model’s context window limit and calibrated to capture 70–90 percent of queries on the fast path (in practice, approximately 90 percent take this path, as shown in the Results section). Because the execution mode is fixed in this first phase, each downstream phase can optimize for its specific path rather than handling both. The second phase, Bifurcated Analysis, executes the chosen path. The fast path makes a single LLM call with the complete context and streams results directly to the evaluation pipeline. The normal path distributes batches across N parallel LLM invocations (where N is the number of batches produced by Phase 1) that analyze chunks independently to achieve near-linear speedup. The third phase, Conditional Consolidation, operates exclusively on the normal path by synthesizing N independent analyses into a unified response. The consolidation LLM receives partial analyses with the original question and applies conflict resolution heuristics while streaming the final answer. The fast path bypasses this phase entirely because the final response only uses one analysis. Figure 2 shows the complete flow, with both paths converging on the streaming response. In production, this threshold-based routing delivers outsized efficiency. Approximately 90 percent of queries take the fast path (single LLM call, time-to-first-token (TTFT) within a few seconds, total latency typically under 25 seconds). The remaining 10 percent (complex multi-document queries) receive the full parallel batch treatment (approximately 50–75 seconds compared to under 25 seconds on the fast path) commensurate with their complexity. With N=4 typical batches for normal-path queries, the blended cost per query is approximately 1.4 LLM invocations. That is a 72 percent reduction in LLM invocations per query versus always using the multi-pass strategy, while maintaining full quality coverage for complex queries. Teams should calibrate their own threshold to match their workload’s query volume distribution and keep it below the model’s context window limit. Cross-account multi-model failover The adaptive pipeline consolidates information efficiently, but a fast pipeline is only useful if the model behind it stays available. During peak review periods, when thousands of users run their analyses simultaneously, a single throttled request erodes confidence in the tool and drives users back to manual spreadsheet analysis. The standard mitigation, exponential backoff with jitter, proved insufficient. Amazon Bedrock assigns independent quotas per model and per account. Treating each model-account pair as its own capacity space multiplies available throughput without provisioning new infrastructure. Rather than waiting for capacity to free up, we set out to discover capacity that was never contended in the first place. This section shows how treating every model-account pair as its own quota space multiplies capacity without provisioning new infrastructure. That capacity hides in plain sight. Amazon Bedrock quotas are independent across two axes: each model has its own limits, and each AWS account receives its own quota. A 3-model × 3-account configuration therefore provides nine independent quota spaces. At runtime, the system explores this grid by attempting the highest-ranking model first, with accounts randomized to help prevent hot-spotting. The implementation is a custom Strands model provider, a drop-in replacement for the standard BedrockModel that adds transparent capacity expansion while maintaining full compatibility with existing agent code. Figure 3 shows the grid and the cascade path a request follows through it. Three-mechanism coordination Three mechanisms coordinate to make this work. The first is model ranking, which establishes a quality-speed hierarchy. Queries attempt the highest-ranking model first and cascade only when necessary, so users receive the best available model automatically. The second is account-level distribution, which helps prevent hot-spotting through stochastic load balancing. Before each failover attempt, Python’s random.shuffle() randomizes the AWS role ARN list order by creating a copy and shuffling it in place. Load then distributes uniformly across accounts over time without requiring complex traffic-shaping algorithms or centralized coordination. Each request gets fresh randomization, naturally achieving approximately 1/N traffic distribution where N is the number of configured accounts. This maximizes aggregate Amazon Bedrock API quota utilization across the account pool. The third is detection and fast recovery, which avoids backoff entirely. The ThrottlingDetector catches ThrottlingException, ServiceQuotaExceededException, and TooManyRequestsException, immediately attempting the next model-account combination. AWS Security Token Service (STS) AssumeRole obtains fresh credentials in 100–200ms, which is negligible compared to the multi-second delays of exponential backoff with jitter. Over a six-month production deployment across over 4,000 users, the N×M quota exploration (where N is the number of models and M is the number of accounts) absorbed traffic surges and reduced user-visible throttling during peak periods. The degree of improvement scales directly with the number of model-account combinations configured. Load testing using the Locust framework revealed that the system was able to support model requests from over 100 users concurrently for a sustained period without any failed requests. The following snapshot shows the load test details for over 100 user requests for a streaming response from Amazon Bedrock through the application. The key takeaway is that infrastructure did not need to scale. The quota space expanded instead. Real-time streaming evaluation The failover architecture now provides an uninterrupted token stream under quota pressure. However, a reliably delivered response that contains hallucinated revenue figures has only made the problem worse. Availability without accuracy is a liability. Our initial approach implemented quality checks as a sequential post-processing step. It would generate the complete response, run the evaluators, then deliver the validated output. While this achieved high accuracy, time-to-first-content (TTFC) exceeded a minute as users waited for the entire response to generate and evaluate before seeing the output. This tradeoff between accuracy and responsiveness led us to a key question: does the entire response need to be complete before validation can begin? This section presents the parallel evaluation architecture that answers that question and achieves near-zero validation overhead by using the producer-consumer concurrency pattern. Paragraph independence Validating output paragraph N does not require waiting for paragraph N+1 to be generated. This logical independence makes parallel execution possible. It dramatically reduces time-to-first-content (TTFC), which is the elapsed time from request submission until the user sees the first delivered evaluated content. In a sequential pipeline, users wait for all paragraphs to generate and then all paragraphs to evaluate before seeing anything. Our parallel approach validates each paragraph as it’s produced, so users see the first content after only one paragraph generates and evaluates, not after the entire response completes. Only the very first paragraph incurs evaluation latency before the user sees content. Every subsequent paragraph is evaluated concurrently with generation of the next, so evaluation cost is absorbed within the generation window. For each subsequent paragraph, the delivery delay is simply whichever takes longer, either evaluating the current paragraph or generating the next one. In practice, paragraph generation takes on the order of a few seconds, while deterministic checks (weasel words, emoji) complete in tens of milliseconds, providing a wide stability margin that keeps evaluation effectively invisible to users in the common case, even with occasional invocation of expensive LLM-based evaluation. Producer-consumer architecture and performance model The implementation coordinates three asynchronous components. The producer task receives tokens from the Amazon Bedrock streaming API and accumulates them into paragraph-sized units detected through configurable heuristics (double newlines for markdown boundaries). Complete paragraphs enqueue in a thread-safe bounded buffer with mutual blocking and First-In-First-Out (FIFO) ordering to preserve coherence. A sentinel value signals completion. The consumer task dequeues paragraphs, executes sequential validation checks such as data accuracy verification, and streams approved content as multi-word chunks for smooth perceived delivery. In practice, this steady-state behavior splits into two distinct regimes depending on which evaluators are triggered for a given paragraph. Define λgen as paragraph generation rate and μeval as evaluation rate. Buffer utilization $\rho = \frac{\lambda_{gen}}{\mu_{eval}}$ determines equilibrium behavior. Given the nature of the evaluation methods (whether deterministic or LLM-based), we would expect a bimodal ρ distribution rather than a single average value. On the fast path, paragraphs requiring only deterministic checks (Weasel Word, Emoji) complete evaluation in approximately 79ms median, yielding ρ ≈ 0.095, far below 1. In this regime, the buffer stays nearly empty because consumers wait for generation, and evaluation adds zero perceivable latency. On the slow path, paragraphs triggering LLM-based numerical verification take approximately 2,025ms to evaluate, yielding ρ median ≈ 21.5, well above 1. Here, backpressure naturally rate-limits the producer, helping prevent unbounded queue growth while the consumer catches up. No content is lost and delivery resumes once the paragraph clears. This split is a direct consequence of the composite evaluation framework discussed later in this post. Most prose clears deterministic checks in milliseconds, while paragraphs containing numerical metrics trigger deeper LLM-based verification. Streaming evaluation results To validate this architecture, we measured performance across 1,000 production queries (approximately 10,439 paragraphs) in an enterprise deployment environment using Anthropic Claude Sonnet on Amazon Bedrock with business intelligence queries averaging over 10 paragraphs per response. Absolute timing values are specific to this environment, but relative improvements and architectural patterns generalize broadly to streaming LLM deployment. The parallel architecture achieves an 86.8 percent latency reduction versus sequential evaluation. The 1.4 second gap versus unevaluated streaming reflects only the one-time first-paragraph evaluation cost before the pipeline reaches steady state. Composite evaluation framework The streaming pipeline delivers paragraphs to an evaluator consumer, but a single monolithic validator addresses only one failure mode. Enterprise quality assurance requires coverage across multiple orthogonal dimensions simultaneously. Responsible AI in this system is therefore two-sided: Amazon Bedrock Guardrails screens every incoming query before generation begins, and the composite evaluators screen every outgoing paragraph before it reaches the user. Language objectivity, formatting standards, and numerical accuracy each represent distinct failure modes that a single evaluator cannot cover. This section presents a composite evaluation framework that coordinates independent evaluators through a shared interface. A formal extensibility constraint keeps the pipeline compatible with the real-time streaming architecture described earlier. Extensibility The composite framework is designed for ongoing extension. As established in the Real-Time Streaming Evaluation section, each evaluator contributes to the pipeline buffer utilization ρ, the ratio of evaluation latency to generation time. Adding a new evaluator is valid as long as the combined ρ across all evaluators remains below 1, keeping the streaming pipeline stable. In practice, cheap deterministic evaluators running in sub-millisecond time can be added unconditionally. Expensive LLM-based evaluators must trigger a sufficiently small fraction of paragraphs to keep the weighted contribution bounded. Each evaluator implements a standard interface. It accepts paragraph content as input and returns a pass/fail result with detected issues and suggested corrections. This supports autonomous operation and pipeline-agnostic integration. Teams can add domain-specific evaluators such as toxicity detection, compliance policy checking, or currency format validation without modifying the pipeline, provided the ρ admission constraint is satisfied. Three reference evaluators and auto-correctors We instantiate three evaluators targeting orthogonal failure modes in business intelligence responses. WeaselWordEvaluator enforces objective language through regex pattern matching because executive reports require quantitative statements (“16 percent growth”) over subjective characterizations (“strong growth”). EmojiEvaluator maintains formal presentation by filtering conversational emoji through Unicode detection. DataAccuracyEvaluator addresses numerical hallucinations, the highest-risk failure mode in business intelligence systems. Fabricated metrics propagating into decision workflows present critical compliance exposure that necessitates two-stage hierarchical verification for comprehensive validation. The evaluators execute in parallel against the same paragraph, so the total evaluation delay equals the latency of the slowest triggered evaluator. A critical violation from an evaluator triggers paragraph rejection. Correctable issues invoke the paired corrector. Universal passage approves streaming to the user. Auto-correction transforms detection into actionable remediation. The pipeline applies correctors in sequenced order. - WeaselWordCorrector removes subjective adjectives while preserving linguistic flow. - EmojiCorrector strips Unicode characters. - DataAccuracyCorrector annotates fabrication with “LLM Reasoning” labels to maintain transparency. The sequencing matters. Text modification runs first, then character removal, then annotation, which maintains stable input for each transformation. Composite evaluation results In production, the composite framework flags substantially more paragraphs than an individual evaluator operating alone. Crucially, nearly a quarter of flagged paragraphs triggered two or more evaluators simultaneously, meaning a single-evaluator deployment would have missed the co-occurring failure in a second quality dimension entirely. This multi-dimensional coverage is the primary benefit of the composite approach. Teams extending this framework can plug in new evaluators with a single interface implementation and no pipeline changes as long as total evaluation time stays within the paragraph generation window. Data accuracy verification Of the three evaluators introduced in the Composite Evaluation Framework section, DataAccuracyEvaluator addresses the highest-stakes failure mode. A response that looks grammatically sound and objectively worded can still hallucinate the exact number that an executive will act on. In practice, this means reporting “$441M” when the actual figure is “$414M,” a discrepancy large enough to drive incorrect resource allocation decisions before anyone catches the error. This failure mode requires a dedicated two-stage verification architecture that cascades from exact value matching through semantic verification. This section presents that architecture, which balances detection accuracy against computational cost. The first stage of this verification performs exact matching to filter ungrounded outputs, while the second stage applies contextual verification through progressive validation. It performs string similarity analysis followed by LLM-based semantic verification. Hierarchical verification Hallucination detection presents competing objectives. Comprehensive accuracy requires semantic understanding, yet real-time deployment demands sub-second validation. Monolithic LLM verification achieves accuracy but incurs prohibitive latency of 2–3 seconds per paragraph. Pure pattern matching provides speed but sacrifices contextual understanding. For example, it can correctly extract “$414M” yet fail to detect inappropriate semantic placement. The two-stage architecture resolves this through cascading validation with progressive computational investment. The design addresses distinct error modes requiring different detection strategies. The first error mode is value fabrication, which occurs when LLMs generate metrics absent from the source corpus. For example, reporting “$441M” when documents contain only “$414M” or citing “Q3 2024 revenue” when source covers only H1 2024. These errors admit deterministic detection through exact value matching in the first stage. The second error mode is context mismatch, which occurs when LLMs misapply source-present metrics, such as using revenue figures for cost discussions, applying Q1 growth rates to Q3 queries, or conflating metrics across business units. These errors require semantic verification in the second stage. The hierarchical approach applies appropriate computational effort to each error class. It uses cheap exact matching for obvious fabrications and expensive semantic analysis only where necessary. Here’s how the two-stage cascade works in practice. The first stage, Exact Value Matching, extracts numerical metrics from both the source documents and the LLM response using regex patterns, then checks whether each metric in the response has an exact match in the source set. Metrics without matches are classified as UNVERIFIED, terminating validation for fabrications at minimal cost. The second stage, Progressive Contextual Verification, operates on matched metrics through two sub-stages. Stage 2a applies SequenceMatcher computing similarity ρ=ratio(Cr, Cs) between response context Cr and source context Cs for w-character windows (w=50). For high similarity, classification proceeds as MATCH, avoiding expensive LLM calls. Stage 2b employs ContextualDataAgent for low-similarity cases (σ<τ), performing semantic verification to detect subtle context mismatches. This cascading design concentrates computational investment where needed. String similarity handles clear matches in approximately 1.7ms per metric while LLM verification addresses ambiguity in approximately 1,758ms per invocation. The computational cost of each stage differs significantly. The first stage takes approximately 0.3ms using regex and set operations. Stage 2a takes approximately 1.7ms for string comparison. Stage 2b takes approximately 1,758ms for LLM inference. Because most metrics pass the first stage (87 percent in production) and only 30 percent of those require the expensive LLM verification in Stage 2b, the effective cost per metric averages approximately 812ms. This represents a 54 percent reduction versus monolithic LLM verification. The reduction comes from routing most metrics through cheap deterministic checks, with only a small portion requiring LLM verification. The degree of cost savings will vary by workload. Teams with higher proportions of numerical content will see more metrics reach Stage 2b, while text-heavy workloads benefit even more from early termination. Architectural integration The five techniques form a layered dependency chain where each layer supports the one above it. The Adaptive Pipeline consolidates variable document volumes into LLM-ready context and routes 90 percent of queries through a single fast pass. This consolidated output reaches the Multi-Model Cross Account Failover layer, which converts hard API quota limits into transparent routing decisions and provides the uninterrupted token stream that the streaming evaluator requires. Without this availability guarantee, the evaluation pipeline must handle both quality failures and generation failures simultaneously, substantially complicating its design. The Streaming Evaluator runs producer-consumer parallelization against this reliable token stream and keeps validation invisible to users in the steady state. The Composite Framework and Data Accuracy Cascade plug into the consumer task, adding three-dimensional quality coverage and numerical hallucination detection without disrupting the latency budget. At the system level, in production deployment, the coordinated architecture delivers responses that start streaming in approximately 13 seconds and maintain 99.3 percent numerical accuracy, with sustained availability across every request observed in this six-month deployment, without a dedicated validation infrastructure, quota increases, or serialization bottlenecks. Conclusion: Bridging AI research and production reality When executives ask data-driven questions in front of their leadership teams, accuracy and speed are non-negotiable. A wrong metric or a delayed response carries immediate professional consequences. Meeting this bar required solving fundamental engineering challenges that go far beyond model selection and basic system design: hallucination of metrics, API throttling under peak concurrent load, validation latency that blocks real-time delivery, and subjective language that undermines credibility. The quality assurance mechanism works as a coordinated pipeline. Adaptive Pipeline Orchestration routes 90 percent of queries through a single fast pass, keeping latency low for the common case while preserving full coverage for complex multi-document queries. Cross-Account Multi-Model Failover can substantially reduce the likelihood that generation stalls by expanding available quota across multiple independent model-account spaces, absorbing traffic surges without user-visible throttling. Real-Time Streaming Evaluation validates each paragraph the moment it is produced rather than waiting for the full response, cutting perceived evaluation latency by up to 86.8 percent. The Composite Evaluation Framework runs multiple independent checks in parallel against each paragraph, catching co-occurring failures that a single evaluator would miss. Finally, Data Accuracy Verification applies a two-stage cascade that catches numerical hallucinations through cheap exact matching first and expensive semantic verification only when necessary, achieving approximately 99 percent numerical accuracy at an average cost 54 percent lower than monolithic LLM verification according to testing. The result is a system that starts streaming validated responses in approximately 13 seconds, remained available across every request observed during peak global load in this six-month deployment, and produces output accurate enough to go directly into an executive business review. For the thousands of leaders who use NarrateAI daily, these techniques are the reason numerical errors have not surfaced in the reviews, answers arrive during the meeting rather than after it ends, the system remained available across time zones throughout the deployment window, and output goes directly into a business review without a rewrite. Next steps To apply these patterns, start by profiling your query volume distribution to determine whether adaptive routing can reduce your LLM costs (Adaptive Pipeline Orchestration). If you are hitting throttling limits during peak load, cross-account failover (Cross-Account Multi-Model Failover) delivers immediate capacity gains with no infrastructure changes. For teams already streaming LLM output, adding producer-consumer evaluation (Real-Time Streaming Evaluation) is the highest-impact quality improvement. For responsible AI, adopt both halves of the control pair described earlier: Amazon Bedrock Guardrails on incoming queries and grounding checks on outgoing responses. Input screening alone leaves hallucinated content unfiltered, and output