loading
loading
A generative AI prototype can answer questions, generate SQL or produce charts. But deploying it in an enterprise is a different challenge. The system must work with live data, enforce access controls, manage unreliable outputs and handle thousands of queries without compromising performance. At Cypher 2026, held in Bengaluru from October 7 to 9, Bhanu Prasad Mahanti and Sri Vallabha Deevi of Tiger Analytics discussed why moving from proof of concept (POC) to production demands a broader engineering effort. From POC to Production Mahanti described three categories of generative AI systems: agentic workflows with deterministic orchestration, single-agent systems that select and use tools, and autonomous multi-agent systems. He says the first two are relatively easier to productionise, while autonomous multi-agent systems are considerably harder. Citing a LangChain developer survey, Mahanti identified output quality as the leading challenge, followed by latency and security concerns. The acceptable level of accuracy also varies by application. Email personalisation may tolerate occasional errors, whereas other enterprise use cases demand much higher reliability. Teams may also need to build their own evaluation datasets. Production introduces further constraints around unbounded inputs and outputs, memory, latency and cost. Mahanti notes that guardrails are essential in production environments because, even if testing appears successful, actual performance remains unpredictable. Working With Live Data Deevi described a deployment for an electronics device manufacturer that collects device-health data in the cloud. The client had a platform where IT administrators could view summaries and run SQL queries, but not all were proficient in SQL. It wanted natural-language interaction with existing data, insights across multiple tables and flexibility as new tables were added. Tiger Analytics integrated an agent-based chatbot into the platform. The system supports multi-turn conversations, generates visualisations and lets administrators investigate data to identify root causes. The engagement has progressed through three versions, with the third being deployed, Deevi says. Tiger Analytics has also developed similar systems for retail and insurance claims. Data freshness posed another challenge. Different tables were updated at different frequencies, with some device data arriving every four hours. A single unified table could not reliably serve every query. Instead, the system identifies user intent, selects relevant tables, generates and executes SQL, and presents insights or visualisations. An earlier iteration achieved 90% accuracy with 10-second latency for complex queries, Deevi says. Privacy Becomes Architecture Production systems cannot always rely on the sanitised data used in prototypes. Personally identifiable information can appear in prompts, databases, charts and vector stores used for retrieval. “You'll encounter them in databases where the device data is stored. You will encounter them in the charts that people will talk about. You may encounter them in the vector stores,” Deevi says. Tiger Analytics replaces PII in prompts with reversible tokens before sending text to a model. Other safeguards include dynamic database masking and sanitising information before embedding it for retrieval. The company also uses small language models to detect attacks and deterministic rules to clean content from external sources. Threats include prompt injection, system prompt extraction, data poisoning and privilege escalation. Outputs from LLM-based tools are treated as untrusted, with human review available for high-impact applications. Access controls must also prevent users from viewing information outside their roles. “The marketing department should not really be seeing sales data. Operations guy should not be seeing HR data,” Deevi says. Role-based permissions and tenant IDs can restrict access. Stricter deployments may use separate databases or vector namespaces, alongside encryption and network isolation. These restrictions cannot depend on prompts alone. “System prompts are always soft instructions. They do not really impose hard restrictions on the data,” Deevi says. When AI Gets It Wrong Connecting a model to enterprise data does not eliminate hallucinations. Errors can occur during query generation, data retrieval or summarisation. “There can be hallucinations at the query level, data level, or the summarisation level,” Deevi says. For customer-facing systems, the consequences can be immediate. For instance, a chatbot offering an incorrect 90% discount could create a business liability. Tiger Analytics restricts query generation to fields in database schemas, attributes sources, enforces output structures and uses fallback mechanisms. Evaluation must continue after deployment. Deevi described using golden datasets, schema validation, small models for toxicity checks and an LLM judge to assess a sample of outputs. Sampling helps contain costs because evaluating every interaction with another LLM can become expensive. Deevi noted that the evaluation model cannot assess every response, raising questions about sampling strategies and the percentage of queries sent to a judge. User behaviour offers another signal. Abandoned conversations, ignored charts and negative feedback can indicate that an answer failed to meet expectations. Production Changes Architecture Moving beyond a prototype also requires rethinking system architecture. Deevi explained that while POCs rely on sequential monolithic designs, production systems must handle thousands of queries per minute. At scale, each request may involve multiple model calls, tools and database operations, creating several potential points of failure. Production systems therefore need asynchronous queues, routing, distributed tracing, failure handling and session-memory management. “So how do you break it into asynchronous queues? How do you gain time? How do you make sure that you handle failure, LLM failures, routing? All these are things that we also need to consider,” Deevi noted. As enterprises embed generative AI into internal and customer-facing systems, the models and frameworks will continue to change. The underlying requirements, however, remain: secure data, reliable outputs, continuous evaluation and resilient architecture. A successful prototype demonstrates what AI can do. Production engineering determines whether it can deliver that value consistently.
document fetched 2026-10-24 11:25