Natural language security queries at Orca Security using Amazon Bedrock

extract

Artificial Intelligence Natural language security queries at Orca Security using Amazon Bedrock Security analysts need fast answers about their cloud infrastructure without deep familiarity with domain-specific query languages. Orca Security’s tool covers over 900 cloud models (the resource types Sonar tracks, such as an Amazon Elastic Compute Cloud (Amazon EC2) instance or an Amazon Simple Storage Service (Amazon S3) bucket). Querying that data required precise model names, nested relationship navigation, and selection from 18 operators (comparison functions like equals or contains used to filter results). To solve this, Orca Security partnered with AWS to build Discovery AI, a natural language query agent on Amazon Bedrock. Orca provides context-aware security for cloud, code, and AI, and Sonar is its domain-specific query system where users search, filter, and explore across the entire data graph. In this post, we share the architecture and results of Discovery AI, which converts a natural language question into a Sonar query. As a result, Orca reduced query costs by 90 percent, input tokens by 93 percent, and latency by 75 percent compared to the initial baseline. Bridging natural language and structured queries Orca Security’s Sonar query system supports 900+ cloud infrastructure and application models with complex relationships between them. To query this data, users construct JSON objects that specify exact model names, select from 18 operators, and navigate parent-child hierarchies across cloud providers. Doing this well requires deep familiarity with the Sonar schema. To build this capability, the Orca team set out to build a natural language interface that can generate valid Sonar queries from plain text. Their initial proof of concept loaded the full Sonar schema into a single prompt on the largest available model. This included over 900 model definitions, every relationship type, and a generous set of example queries. The output quality was strong (design partners and internal users responded positively) but three significant drawbacks emerged at production scale. Token consumption was inefficient, at roughly 2 million input tokens per workload, which drove up cost. Latency was high, with end-to-end response times of around 60 seconds per workload. Accuracy on complex, multi-model queries was limited because nested relationships and multi-condition queries often required additional refinement. The Orca team validated that the concept worked. The next challenge was making it cost-effective, fast, and accurate enough for production. To address this, the Orca team designed and deployed a solution that handles thousands of queries per day at 15-second response times using Amazon Bedrock. Solution overview Orca designed Discovery AI as a hybrid solution on Amazon Bedrock that combines Retrieval Augmented Generation (RAG) with schema-aware query construction. The system converts natural language questions into valid Discovery filters through a multi-stage pipeline, with each stage matched to the most appropriate Amazon Bedrock model. Orca chose Amazon Bedrock because it provides multi-model access through a single API, letting the team allocate different foundation models to each pipeline stage without managing separate integrations. Amazon Bedrock also inherits AWS governance controls through AWS Identity and Access Management (IAM), virtual private cloud (VPC) endpoints, and AWS CloudTrail, meeting Orca’s security requirements. The model-agnostic architecture meant the team could swap in newer model versions as they became available without rearchitecting the pipeline. The following diagram illustrates the nine-stage pipeline architecture. The Discovery AI pipeline converts a natural language request into a validated Discovery filter through nine steps. - Request Entry and Context Setup: The pipeline starts by combining the user’s natural language request with relevant context, including the user’s organization, connected cloud providers, and request metadata. This context helps the pipeline interpret the request accurately and generate a Discovery filter that matches the customer’s environment. - Quick Deterministic Matcher: The Quick Matcher identifies structured inputs such as IP addresses, Common Vulnerabilities and Exposures (CVEs), ARNs, file hashes, Amazon Machine Image IDs, Oracle Cloud Identifiers, EC2 IDs, URLs, and alert IDs. It uses regex and rule-based matching to detect high-confidence patterns.When a match is found, the pipeline can return a predefined Discovery filter through a fast path, without using large language model (LLM) inference. When no match is found, the request continues to the next step. - LLM Guardrail Screening: The Guardrail component checks whether the request is safe, relevant, technical, and supported. It blocks prompt injection attempts, off-topic requests, and unsupported inputs before they reach the rest of the pipeline. Requests that fail the guardrail checks are rejected with an appropriate error. Approved requests continue to example retrieval. - Embedding-Based Semantic Candidate Recall: The pipeline searches for relevant previously validated query examples using Amazon Titan Text Embeddings. It retrieves the 20 nearest examples from the vector store based on semantic similarity. These candidates provide the pipeline with examples of how similar natural language requests were previously translated into Discovery filters. - Neural Relevance Reranking: The retrieved candidates are reranked using Cohere Rerank 3.5 on Amazon Bedrock. It identifies the examples that are most relevant to the current request and selects the five highest-signal examples. When an exact or high-confidence match is found, the pipeline can reuse the known-good filter directly. Otherwise, the reranked examples are passed to the generative stages. - LLM-Driven Relevant Model Selection: Model Selection narrows more than 900 Sonar models to the small set relevant to the request. It considers the user’s request, connected providers, and retrieved examples when selecting the appropriate models. The selected models are provided to the Generator using the appropriate schema level, ranging from compact micro schemas to more detailed mini schemas. This reduces unnecessary context and makes generation more efficient. - Tool-Augmented Agentic Filter Generation: The Generator uses a tool-augmented agent to construct the final Discovery filter. It reasons over the selected model schemas, retrieved examples, user context, and available tools. The agent can iteratively refine the filter as it works toward a valid representation of the user’s request. - Static and API-Backed Validation: The generated filter is checked for syntactic and semantic correctness. The pipeline first applies static validation and then verifies the filter against Orca’s Sonar API. When validation fails, the validation error is returned to the Generator so the agent can correct the filter and try again. If the maximum number of retries is reached without producing a valid filter, the pipeline returns an invalid-filter response. The Generator is constrained to the reranked examples and selected schemas. Static validation resolves every model, field, and operator back to those schema definitions. Each validation error goes back to the agent with the offending element identified, so retries re-ground rather than re-guess. - Post-Processing and Response: After a valid filter is produced, whether through the Quick Matcher, reuse of a known-good example, or agentic generation, the pipeline prepares the final response.The team allocated different foundation models on Amazon Bedrock across components based on each stage’s requirements. Anthropic Claude Haiku 4.5 powers the Model Selector and Generator. These components require strong reasoning for accurate query construction. Amazon Nova 2 Lite handles the Guardrail and Validator, where fast inference at lower cost is prioritized. Amazon Titan provides embeddings for semantic search in the Example Retrieval component. Cohere Rerank 3.5 reranks retrieved candidates to maximize relevance. For model availability by AWS Region, refer to Regional availability by models. Maximizing relevance and diversity in example retrieval Example retrieval emerged as the single biggest accuracy lever in the pipeline. Strong examples teach the Generator how to translate questions into Sonar queries more efficiently than schema alone, so the Orca team benchmarked several retrieval strategies on Amazon Bedrock. Each strategy traded off relevance, diversity, or speed in a different way, and comparing them directly showed which combination worked best. Pure top-K semantic search on Amazon Titan embeddings performed quickly, but it tended to return near-duplicate examples that starved the Generator of diversity. Maximal Marginal Relevance (MMR) improved diversity, but it traded off relevance when the top candidate was actually the right answer. Retrieving a larger candidate set and then reranking it pulled in a broader pool and used a reranker to select the most relevant subset, which pointed toward the winning approach. The winner was top-20 nearest-neighbor retrieval using Amazon Titan embeddings. Orca then reranked these results with Cohere Rerank 3.5 down to the top five. This approach gives the Generator a small, diverse, highly relevant set of examples without forcing a trade-off between coverage and precision at retrieval time. Loading over 900 models into context A core constraint shaped the architecture. It wasn’t feasible to include 900+ Sonar model schemas into every prompt. Even with long-context models, doing so degrades both cost and accuracy. The Model Selector resolves this. Running on Anthropic Claude Haiku 4.5, it works against compact, semantically rich representations of each model (name, purpose, key fields, and key relationships). Given the user’s question, it returns only the models the Generator actually needs. The Generator then receives full schema detail only for those selected models plus the relationships that connect them. Evaluation methodology To iterate confidently, the Orca team built a continuous evaluation harness early in the project. The harness combines a curated dataset, an automated scoring layer, and a slice-based measurement approach so the team can trust every change before it ships. Together these elements turn evaluation into a routine step rather than an afterthought. The curated evaluation dataset spans the question types, model families, and complexity tiers seen in production, so it reflects real usage. An LLM-as-judge scoring layer grades generated queries on correctness, completeness, and semantic equivalence to a reference query, which gives the team a consistent quality signal. The measurement methodology then breaks deltas down by slice, allowing the team to identify exactly where regressions occur and what to correct. Every change to a prompt, a model, a retrieval parameter, or an example flows through the harness before it ships to production. Domain-expert-in-the-loop labeling system for continuous improvement Sustaining accuracy as customer environments evolve and new question patterns emerge required more than a strong initial deployment. The Orca team built an internal labeling tool for product managers, support engineers, and security researchers. With the tool, they can flag queries, correct outputs, and push corrections into the example bank in one action. Furthermore, the system automatically collects production queries with low-confidence matching examples and routes them to the appropriate domain expert for daily labeling. The next pipeline run retrieves the new example through Cohere Rerank 3.5, and accuracy on that class of question improves the same moment, with no deployment. “The labeling UI is the single most important investment we made for long-term quality. It turns Discovery AI from a static deployment into a living system, with our domain experts continuously improving it inside their normal workflow.” — Or Gabay, Director of AI, Orca Security Combined with end-to-end pipeline observability and shared dashboards spanning product, engineering, and customer-facing teams, this feedback loop is what keeps Discovery AI improving after initial launch. Results After deploying Discovery AI to production, the new solution outperformed the initial baseline across cost, latency, and accuracy. Input token usage fell by 93 percent, from approximately 2 million tokens down to 140,000 per comparable workload. Overall cost fell by 90 percent, driven by that lower token consumption together with per-stage model allocation across Amazon Bedrock. Latency also improved, with end-to-end response times dropping by 75 percent, from approximately 60 seconds to 15 seconds. Accuracy on complex, multi-model queries also improved, measured continuously through the team’s evaluation harness. Discovery AI now handles thousands of queries per day in production since March 2026, with internal subject matter experts and customer feedback validating quality. Conclusion In this post, we shared how Orca Security built Discovery AI on Amazon Bedrock. The solution reduced query costs by 90 percent, input token usage by 93 percent, and latency by 75 percent compared to the initial baseline. The solution combines several key technologies. These include RAG-based example retrieval with Cohere Rerank 3.5, per-stage model allocation across Anthropic Claude Haiku 4.5 and Amazon Nova 2 Lite, and semantic search powered by Amazon Titan. Additionally, a continuous human-in-the-loop labeling system provides ongoing improvement. This combination allowed Orca to turn a working prototype into a production-grade solution that is both cheaper to run and more accurate. To get started building similar solutions, explore Amazon Bedrock for foundation model (FM) access, Cohere Rerank on Amazon Bedrock for retrieval optimization, and Amazon Titan for embeddings. To learn more about Orca Security’s cloud security tool, visit orca.security. To learn more about how Orca Security applies AI across its framework, see how Orca drives security outcomes with AI and how Orca secures AI adoption, including Amazon Bedrock. For more on the techniques used in this post, read related posts on the AWS Machine Learning Blog.