loading
loading
Artificial Intelligence How Rescale built an AI triage agent with Strands Agents and AI-DLC Every engineering team knows the ritual, and Rescale was no exception. A nightly test suite fails. An on-call engineer pulls the logs, classifies the root cause, and routes it to the right team, only to repeat the process the next morning. At scale, across distributed services and dozens of repositories, this ritual becomes a full-time job hiding inside an on-call rotation. Rescale is the digital engineering platform built for the AI era. R&D and engineering teams across aerospace, automotive, energy, life sciences, semiconductor, and manufacturing use the Rescale platform to accelerate innovation, bringing intelligent HPC, advanced modeling and simulation, agentic digital engineering, and AI physics together on a single platform. Beyond coding, Rescale uses AI across the engineering lifecycle, from planning and design through implementation, testing, review, and operations, while keeping engineers accountable for the result. AI can propose and execute, but people define scope, validate the work, and approve production changes. Engineers can then focus on architecture, product judgment, and the context required to operate complex distributed systems. In this post, we show how Rescale used the AI-Driven Development Lifecycle (AI-DLC) framework on AWS to build an intelligent triage agent. They went from problem definition to staging deployment in five days, and to production within weeks. The post also covers how Rescale is extending AI-DLC across their engineering organization. The challenge As Rescale’s platform grew, so did the operational overhead of keeping it running. Engineers were spending significant time on a repetitive but critical task: classifying automated end-to-end test failures. Hundreds of test cases run in the suite, and when failures appeared, every alert, bug report, and escalation needed to be classified, prioritized, and routed to the right team. After routing, engineers would need to gather evidence from multiple logging and monitoring systems, along with several source repositories before debugging could even begin. Passing end-to-end tests were required to deploy services to production and thus debugging these system failures was in the critical path to getting new changes to production. In the manual workflow, engineers moved from the test report to logs, historical patterns, and source code to classify failures and identify the likely owning team. Because these integration tests gate weekly production releases, unresolved failures could pull several teams into hours of coordination before a deployment was allowed to proceed. The team had a clear idea of what a solution looked like: an intelligent agent that could classify and route issues automatically. What they were looking for was a structured way to go from that understanding to a working system in production, with enough structure to extend the approach to other use cases afterward. Why Rescale chose AI-DLC Rescale’s triage agent was a well-scoped problem with a tight timeline. The team needed a development approach that could move at that pace without skipping the architectural decisions that would determine whether the system worked in production. The AI-Driven Development Lifecycle (AI-DLC) is an open source, AI-centric methodology for software development built around two ideas: - AI executes, humans decide: AI generates work plans, writes code, and proposes architecture, but it surfaces decision points and asks clarifying questions rather than guessing. Humans validate and steer at each step. - Team sessions replace async handoffs: Instead of one person building alone and requesting review later, AI-DLC uses focused group sessions: Mob Elaboration (defining requirements together) and Mob Construction (building together while AI handles implementation). Decisions happen in real time. AI-DLC fit because the hard part wasn’t writing code. It was making sure the AI understood what the team wanted to build. In mob sessions, Rescale engineers answered the AI’s questions together and approved each plan before the AI moved on, so nothing was left to assumptions. Solution overview Rescale’s triage agent automates the classification and routing of nightly end-to-end test failures. It runs as a step within Rescale’s existing Argo Workflows continuous integration and continuous delivery (CI/CD) pipeline on Amazon Elastic Kubernetes Service (Amazon EKS). When a nightly Playwright suite of hundreds of test cases produces failures, the pipeline triggers the triage agent pod. The agent performs the following steps: - Invokes Claude Opus 4.8 on Amazon Bedrock to classify each failure into a root cause category: infrastructure issue, code bug, flaky test, or environment problem. - Retrieves relevant historical failure patterns from a knowledge base for additional context. - Queries pod logs through Model Context Protocol (MCP) for runtime evidence supporting the classification. - Generates a structured remediation report with specific file references, function names, and recommended fixes for engineer review before any action is taken. The agent completes this workflow in seconds and outputs directly into the team’s existing reporting pipeline. The following diagram shows how these components fit together: The AI-DLC workshop experience The engagement was structured as a five-day, in-person workshop. AWS architects guided the build while Rescale engineers worked hands-on-keyboard, co-building in real time. Inception phase (days 1–2): From vision to validated requirements The team started by scoping requirements, capturing the intent in a vision document and a technical environment document that grounded the work before any building began. That intent began as a single statement, “We want AI-powered test triage that classifies failures and recommends remediation”. Working collaboratively in mob format, the team answered 20+ clarification questions generated by AI. This produced a scoped requirements document with Phase 1 boundaries explicitly defined and Phase 2 items deferred. Five user stories were consolidated into four units of work: - Knowledge base for historical failure patterns. - AI triage agent as the core classification and remediation engine. - Argo Workflow Integration for trigger, infrastructure, and deployment. - Evaluation Harness for accuracy measurement and regression gating. Human decisions drove the critical boundaries: what belongs in Phase 1 versus Phase 2, priority ordering, and deferred features like automated pull request (PR) generation. Construction phase (days 2–4): Architecture to running code The team defined interface contracts and data schemas, completed functional and non-functional design questions, and signed off on the architecture. Two architecture pivots emerged during the build, and the team validated each one before changing course. First, the team planned to host the agent on Amazon Bedrock AgentCore, then moved it to Amazon EKS, where Rescale already runs its Argo Workflows CI/CD pipeline. Keeping the agent in the existing cluster meant no new runtime to operate and a direct trigger path from Argo. Second, in the early build the agent pulled log evidence from a third-party logging service through MCP. The team moved log retrieval to Amazon CloudWatch, which simplified the dependency chain. The MCP pattern stayed the same. Only the source behind it changed. Because AI-DLC keeps requirements and design decisions in shared documents, the team updated them in place and kept building without losing context. The agent was built using the Strands SDK, supporting concurrent failure analysis with structured output in JSON and HTML, containerized on Amazon EKS. By Day 4, the full pipeline was running in staging: Argo workflow trigger, triage agent pod, MCP log query, knowledge base retrieval, and structured triage output. Operations phase (day 5): Evaluation and handoff The team built a PyTest-based evaluation framework with live and stubbed test modes, establishing an initial classification-agreement baseline to measure the agent against and improve from. The agent was demoed live to Rescale’s CTO and VP of Engineering, correctly classifying both an infrastructure failure and a real code bug with specific file-level fix recommendations. How AI-DLC scaled rigor to complexity Not every unit of work gets the same treatment. The knowledge base needed minimal design and deployed using Terraform in hours. The Triage Agent got full rigor: architecture review, interface contracts, 12 design questions, and iterative code review. Argo Integration focused on infrastructure contracts and IAM. The Evaluation Harness used lightweight design but heavy implementation. AI-DLC matched depth to complexity automatically. By the end of Day 5, the agent was live in staging, processing real test failures with a measured accuracy baseline and a clear path to production. Production outcomes The triage agent has been running in production since the workshop. It automatically classifies nightly end-to-end test failures, replacing manual first-pass triage with an automated first pass that engineers validate. For each failure, the agent generates a structured remediation report with actionable next steps rather than raw logs. The agent’s classification agreement with human triage has risen from 44% at initial deployment to 68% today. It continues to improve as the team iterates on the golden dataset and prompt engineering. Since the triage agent entered production, the team has measured a 20% decrease in time blocking production deployment while test failures are analyzed. Because these tests gate Rescale’s weekly production releases, that reduction shows up directly in how quickly changes reach production. Engineers also spend less of the on-call rotation on repetitive classification. Life after the workshop: From one agent to an AI-first practice The workshop delivered one production agent, but its larger effect was on how Rescale approaches engineering work. The triage agent is part of a broader push at Rescale to deliver digital engineering capabilities for the AI era. Internally, Rescale is applying these same benefits realized from AI across its own engineering organization. AI-DLC gave Rescale’s engineers a structured, repeatable way to put that direction into practice. Scaling AI-DLC across the organization Rescale is evaluating an internal AI-DLC Center of Excellence, a small group that codifies best practices and helps other teams adopt the methodology independently. The engineers who participated are already applying what they learned. One of the engineers noted that the process “will be part of my workflow going forward,” not only for building agents, but for any complex development work where AI can contribute. Broader AI-native development Rescale is now incorporating AI-DLC into the standard software development lifecycle for new engineering simulation products. The framework brings simulation-domain expertise, product requirements, platform architecture, validation, and operations into one structured design process. Teams resolve key questions early, capture reviewed decisions in executable specifications, and give coding agents focused work with clear acceptance criteria. This shortens the path from product concept to validated implementation while engineers remain accountable for product correctness. The same approach is already being used in product-enabling platform work and agent systems: - HPC simulation application catalog management. - Container vulnerability tracking and remediation for government compliance requirements. - Workflow-based cloud coding agents. Conclusion For the triage agent, the most useful output of AI-DLC was not the code itself. It was the structured path from an initial idea to a reviewed system design: clarifying questions, scoped requirements, and evaluation cri
document fetched 2026-10-24 11:20 · canonical https://aws.amazon.com/blogs/machine-learning/how-rescale-built-an-ai-triage-agent-with-strands-agents-and-ai-dlc