How HPE Zerto built an agentic troubleshooting system with Amazon Bedrock

extract

Artificial Intelligence How HPE Zerto built an agentic troubleshooting system with Amazon Bedrock This post was co-written by AWS and the HPE Zerto team. If you manage hybrid and multi-cloud infrastructures, you may already be turning to AI systems to assess health, investigate issues, and act on problems faster. HPE Zerto addressed this challenge by building an agentic troubleshooting system powered by Amazon Bedrock. HPE Zerto Software helps you strengthen cyber resilience, disaster recovery, and continuous data protection. You can use these capabilities to substantially reduce data loss and downtime, while supporting rapid recovery from ransomware events, outages, and other disruptions. This post describes the system architecture we selected and the challenges we faced when developing an agentic troubleshooting system powered by Amazon Bedrock. We deployed it in an on-premises environment, with controlled and secure access to operational signals, environmental context, and product knowledge. The system provides a natural language experience that helps you move quickly from questions to actionable insights to support faster, more informed recovery decisions. Business challenge Data protection and disaster recovery teams operate in environments that are growing in scale, complexity, and business criticality. They must continuously monitor protection health, validate service level agreement (SLA) compliance, interpret alerts, investigate failures, and prepare for recovery events. This work often spans multiple sites and large numbers of protected workloads. - Information needed to answer operational questions is often fragmented across product alerts, events, and documentation, forcing teams to manually gather context before they can act. - Operators lose time moving between dashboards, reports, and knowledge sources to understand what is happening and what to do next. - Less experienced administrators may struggle to interpret symptoms or determine the safest remediation path. - Managers may lack fast access to clear summaries of risk, SLA exposure, or recovery readiness. - During outages or cyber events, these delays become even more significant because the cost of uncertainty is high and teams need clear, trusted guidance quickly. Our customers therefore need a way to turn complex resilience data into actionable answers without adding more operational burden while deployed in different environments. Solution overview Fundamentally, we designed the system to: - Serve as a support assistant helping you solve configuration issues and troubleshoot problems in a short-circuit without needing to create a support ticket. - Alert you on system health problems and assist with mitigation. - Perform tasks on your behalf to accelerate setup time and feature adoption. We deploy the system as a pod in the Zerto product inside your on-premises environment, accessible directly from the same UI you already use for day-to-day operations. You can interact in natural language, and the system is designed to reason, act, and respond with contextual, actionable answers grounded in the live state of your environment. We selected Amazon Bedrock because of its ability to support secure enterprise deployment, model flexibility, and operational control. The key considerations included: - Security and governance – Amazon Bedrock supports controlled access to foundation models, helping us align AI usage with enterprise security, compliance, and data protection requirements. - Model flexibility – The service made it possible to access and evaluate multiple foundation models and select the best fit based on quality, latency, and cost. - Operational integration – Amazon Bedrock integrates with AWS services used for guardrails, observability, and knowledge retrieval, supporting a more manageable production architecture. We enforce content security using Amazon Bedrock Guardrails to make sure your prompts and system output are aligned with company policy, compliance, and security requirements. The following figure shows the key components of the architecture: - UI layer – You interact with the system through the chat interface embedded in the existing Zerto UI. Real-time streaming using Server-Sent Events (SSE) shows investigation progress as it happens, providing a better user experience. - Agentic layer – Agents built using the Strands Agents framework run as a pod inside your environment on-premises, with access to the following: - Session history – Locally stored historical chat conversations and context. - Internal tools – Local Model Context Protocol (MCP) server exposing ZVM (Zerto Manager) APIs the agent can call for structured access to live environment data. - External tools – Smart fetching of public documentation, relevant runbooks and operational knowledge using Amazon Bedrock Knowledge Bases, a centralized, fully managed Retrieval Augmented Generation (RAG) capability with internal semantic search for context retrieval. - Intelligence layer – Comprised of three main components: - Inference – Requests are sent to Amazon Bedrock which routes them to the selected foundation model. Per-tenant request tagging is done using AWS Identity and Access Management (IAM) role tags each tenant assumes. - Security – Amazon Bedrock Guardrails are applied before and after inference to enforce content boundaries and help verify responses remain scoped to operational topics. - Limits and quotas – Amazon CloudWatch, Amazon DynamoDB, and AWS Lambda are combined to enforce per-tenant quotas and limits and maintain a cost-efficient system. - Observability layer – Telemetry data sent from the system to Amazon CloudWatch provides visibility into agent performance, error rates, and usage patterns. The following figure shows the agentic system embedded in the Zerto product. Agentic flow The system consists of multiple sub-agents, controlled by an Orchestrator agent which can use each of those sub-agents as tools based on the task in hand. In turn, each sub-agent gets its own set of tools and separate context to handle its sub-task. Figure 3 shows the flow and hierarchy between the agents. The Orchestrator decides when to delegate work. Routine questions are answered directly, while deep, log-heavy investigations are routed to a specialized sub-agent, scoped to the relevant component. Each sub-agent runs with its own fresh context, dedicated system prompt, its own tools, and uses a stronger model for the reasoning-heavy work. Its final response is a compact, distilled report, without raw logs and investigation scratch-work polluting or inflating the parent’s context. - Orchestrator agent – Serves two purposes. You can use it for checking protection group status, reviewing recent alerts, or asking about configuration best practices. When given an issue that currently exists in the system, the agent orchestrates the troubleshooting flow between the sub-agents. It relays the final answer to you after stripping secrets or raw logs from the response. - ZVM agent – Responsible for troubleshooting ZVM related issues such as service crashes, networking issues, and upgrade failures. Using dedicated skills that provide domain-specific knowledge and specific log patterns, it analyzes logs and adds environmental data through MCP. This provides a thorough analysis of the issue at hand and its possible causes. - VRA agent – Responsible for troubleshooting VRA (Zerto Replication Engine) related issues such as replication problems and delays, networking between sites and installations. It also has domain-specific knowledge skills that provide structured log patterns context for the agent to match against the logs and the environmental data available through MCP. Challenges Building the HPE Zerto agentic troubleshooting system required navigating several key engineering decisions, including model selection, evaluation methodology, and the constraints of on-premises deployment. Model selection As part of our development process, we compared and then combined several models for reasoning and inference, considering the tradeoff between quality, performance, and cost. We evaluated bigger, stronger models first for the initial proof of concept (POC), to prove feasibility, and later implemented an agent evaluations mechanism (described in the following section) and compared other models, some of them faster, smaller, and cheaper. Eventually, a multi-model architecture was selected as described earlier. We continue to evaluate newer models as they become available. For model availability by AWS Region, refer to Supported models by AWS Region in Amazon Bedrock. Agent evaluations We used PyTest and Strands Agents Evals SDK for evaluating agents systematically and on an ongoing basis as we keep upgrading the system. We have also created jobs for running the full evaluations suite to see regressions and improvements in the agent behavior. Three types of evaluations were created: - Basic query evaluation – Single-turn query-response evaluation where the agent is given a query as a test and the response is evaluated. - Multi-turn evaluation – Full conversation evaluation where the agent is given an initial query as a test, the agent responds and a user simulator large language model (LLM) acts as an actual user. This technique uses Strands Agents ActorSimulators. The full conversation is evaluated at the end. - Dynamic tests evaluation – To evaluate uncovered and dynamic use cases, we created a fully dynamic test suite using the Strands Agents Experiment Generator, which creates a unique test suite based on specific criteria.For each test there are four evaluators: - Response evaluator – Evaluates the agent response based on specific predefined criteria and using structured output that can be parsed to extract relevant data. - Trajectory evaluator – Evaluates the agent tools selection based on specific pre-defined criteria. This evaluator also uses structured output. - Latency evaluator – Evaluates the agent latency and checks it stands within the threshold defined. - Token evaluator – Evaluates the agent token usage and checks it remains within boundaries. “Strands Agents eliminates the complex work of building AI agents — no rigid workflow definitions, no boilerplate tool-chaining code, just a model, a prompt, and the tools you need. With built-in evaluations and observability, my team builds, tests, and iterates on agents faster than with any framework we’ve tried. It covers the full development lifecycle so we can focus on solving real problems, not writing glue code.” — Shahak Gabay, Senior Software Engineer, HPE Zerto On-premises deployment model One of the constraints around deployment of our agentic troubleshooting system is that it must run inside your own environment. Disaster recovery (DR) infrastructure is frequently air-gapped, latency-sensitive, or subject to strict data residency requirements. We implemented the agent runtime using the Strands Agents SDK, which runs entirely on-premises within the ZVM appliance. Only model inference requests and knowledge base queries go out to AWS over HTTPS. Business outcomes With the HPE Zerto agentic system, you can improve operational efficiency while strengthening recovery readiness. By grounding responses in live environment data and trusted product documentation, the system helps you answer operational questions, investigate issues, identify likely root causes, and prioritize risks that require action. Since its release in Q2 of 2026, more than 20 percent of HPE Zerto customers have already adopted and are actively using the agentic system. This demonstrates strong demand for AI-assisted disaster recovery operations. Across production usage, we have observed a 10 percent reduction in support cases for workflows supported by the agentic system. You can now investigate and resolve more common operational issues directly within the product through guided self-service. The impact extends beyond support-case reduction. By reducing repetitive data gathering and manual investigation, the system helps accelerate your troubleshooting and root-cause identification while improving administrator productivity. You can generate operational summaries faster, identify SLA and protection risks earlier, and make more informed decisions during both routine operations and incident response. As adoption continues to grow, these capabilities can help you reduce operational friction, improve recovery readiness, and scale disaster recovery operations more effectively without requiring a proportional increase in specialized expertise. Internally, our engineers have started using the agentic system’s troubleshooting capabilities to investigate and resolve common operational issues. Moreover, our QA automation team has integrated the agentic system into their workflows, now performing faster and more accurate investigation and resolution of issues found during testing. Lessons learned Building and deploying this solution surfaced several insights that shaped our approach to agentic architectures. These insights center on on-premises constraints, agent decomposition, and real-time user feedback. - On-premises deployment changes the architecture fundamentally. The constraint that no customer data could leave your on-premises network shaped every architectural decision, from our choice of the Strands Agents SDK for local orchestration to the MCP server design for local data access. - MCP servers are an effective pattern for grounding agents in live operational data. Wrapping an existing product API in an MCP server gives agents structured, typed access to live environment data and significantly accelerated our agent development. - Multi-agent decomposition improves reliability for complex tasks. Breaking the workflow into multiple specialized agents, each with a bounded responsibility, made each component independently testable and the overall system easier to debug.We wired the sub-agents in a hub-and-spoke topology. Every delegation flows through the parent, and a sub-agent that discovers the issue belongs elsewhere reports back to the parent rather than calling a peer directly. This design keeps a single point of control for safety rules and the final answer, and it keeps the flow linear and straightforward to trace in telemetry. It also helps prevent runaway agent-to-agent recursion that could exhaust the token budget.The result is better answers on complicated problems without sacrificing the speed and predictability of everyday chat. - Streaming matters for user trust. Showing investigation progress in real time through SSE significantly improved the user experience. Users see each step as it happens rather than waiting without visibility. Conclusion HPE Zerto Software brings together trusted resilience expertise, operational data, and AI-driven assistance, with Amazon Bedrock generative AI service, to help you protect, recover, and operate with greater confidence. By embedding intelligent guidance into everyday protection and recovery workflows, we help you turn complex resilience signals into clear, actionable insights, supporting faster decisions and stronger SLA readiness. With HPE Zerto, you can move beyond point-in-time recovery readiness and build a more intelligent, resilient approach to continuous business protection. To read more and go deeper, explore these related resources: - Evaluating AI agents for production: A practical guide to Strands Evals - Multi-Agent collaboration patterns with Strands Agents - Beyond the basics: A comprehensive foundation model selection framework for generative AI