Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions

extract

Artificial Intelligence Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions Build a continuous integration and continuous delivery (CI/CD) quality gate that deploys an agent with role-based MCP tools, evaluates it, and blocks PRs when evaluation scores drop. You shipped an AI agent on Amazon Bedrock AgentCore runtime. It calls tools through an MCP server protected by OAuth. Now you want CI to tell you when a code change makes its performance worse before it reaches production. This post walks through a GitHub Actions pipeline that deploys an agent to AgentCore runtime and evaluates the agent with evaluation prompts using the AgentCore Evaluate API. If the agent regresses, the PR fails. We’ll cover the full stack: a Strands agent that connects to an MCP server with role-based access control, a shared Cognito pool serving both machine-to-machine (M2M) and user-scoped auth flows, CDK infrastructure-as-code, and a unified evaluation script. The complete reference implementation is available in the accompanying repository. What this covers - Deploying an agent + MCP server to AgentCore runtime using CDK. - Role-based access control on MCP tools (three-layer auth pattern). - Invoking OAuth-protected runtimes from CI (M2M client_credentials flow). - Running on-demand evaluations with built-in evaluators. - Enforcing a quality gate that blocks merges on regression. - Handling the OAuth challenge: how CI pipelines authenticate without user context. Key concepts Before diving in, here’s a quick primer on the building blocks. Skip ahead if you’re already familiar with them. - AgentCore runtime is a managed hosting platform for AI agents. You deploy your agent code (Python, any framework), and AgentCore handles scaling, session isolation, and infrastructure. Think of it as AWS Lambda for agents. - AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, scores agent behavior using a large language model (LLM) as a judge. It reads OpenTelemetry traces from Amazon CloudWatch and rates responses on dimensions like helpfulness, correctness, and tool selection accuracy. - MCP (Model Context Protocol) is an open protocol that agents use to call external tools through a standardized interface. An MCP server exposes tools. The agent discovers and calls them. AgentCore runtime can host MCP servers and connect agents to them. - OpenID Connect (OIDC) federation is how GitHub Actions assumes an AWS Identity and Access Management (AWS IAM) role without storing long-lived credentials. GitHub issues a short-lived token, AWS validates it, and the workflow gets temporary credentials. - Quality Gate is a CI/CD pattern where a pipeline step must pass a threshold before the build can proceed. In our case, the agent’s evaluation scores must meet a minimum bar (for example, 0.8 out of 1.0) or the PR stays blocked. Why this matters: Without automated evaluation, agent quality is subjective. A developer changes a system prompt. The agent starts giving worse answers, and nobody notices until users complain. A quality gate catches this at PR time before it reaches production. The problem Here’s the scenario. You have an agent deployed on AgentCore runtime. It calls tools through an MCP server where some tools are public. Others are restricted by user role. Every time someone changes the system prompt, swaps a model, or updates tool configurations, you want to know: did the agent get better or worse? Manual testing doesn’t scale. You need automated evaluation in CI. That means automatically deploying the agent in a dev environment, invoking it with representative prompts, scoring the responses, and blocking the merge if quality drops. The complication: your MCP server uses OAuth with role-based access control. CI pipelines don’t have user context. How do you authenticate a headless pipeline against an OAuth-protected agent that forwards tokens to an MCP server expecting user roles? AgentCore Evaluations: Where it fits AgentCore Evaluations is the quality measurement layer in the Amazon Bedrock AgentCore platform. It sits alongside AgentCore runtime, which hosts your agent, and AgentCore Observability, a capability of Amazon Bedrock AgentCore that captures traces, completing the build → deploy → observe → evaluate lifecycle. The service scores agent interactions using LLM-as-a-judge by default, with an option for code-based evaluation via AWS Lambda. It operates on OpenTelemetry traces, the same traces your agent already emits through AgentCore Observability. For on-demand evaluation, you provide span data directly in the API call; online and batch evaluation read from CloudWatch. Three evaluation modes cover different stages: - On-demand evaluation evaluates specific sessions at any time. You provide span data, pick your evaluators, and get scores back. This is what powers CI/CD quality gates, the focus of this post. - Online evaluation continuously monitors production traffic with configurable sampling rates. Results feed into CloudWatch dashboards for trend monitoring. - Batch evaluation scores multiple sessions in a single asynchronous job. You point it to your CloudWatch Logs, pick your evaluators, and get aggregate plus per-session results. This is what powers baseline measurement and pre/post regression testing. Evaluators come in four categories: - Built-in evaluators cover common quality dimensions: Helpfulness, Correctness, GoalSuccessRate, ToolSelectionAccuracy, ToolParameterAccuracy, and more. They operate at session, trace, and tool-call levels. Three trajectory evaluators (TrajectoryExactOrderMatch ,TrajectoryInOrderMatch ,TrajectoryAnyOrderMatch ) compare actual tool-call sequences against expected trajectories. - Custom evaluators use your own LLM-as-a-judge prompts for domain-specific scoring. Ground truth fields (expectedResponse, assertions, expectedTrajectory) are available as placeholders in custom evaluator prompts too. - Code-based evaluators run a Lambda function against each trace or session and return a score, label, and explanation calculated by your custom implementation. Use them for deterministic checks like regex matching, schema validation, or keyword presence without LLM costs. - Third-party evaluators from the DeepEval and AutoEval open-source libraries are managed by the service like built-in evaluators. Select one by ID with no model or configuration required. You can also derive a custom evaluator from a built-in or third-party evaluator to run its logic on your own model. The Evaluate API accepts sessionSpans (OpenTelemetry trace data from CloudWatch) and returns structured scores. Each evaluate() call must contain spans from a single session only. Mixing sessions causes a ValidationException. The API also accepts optional ground truth through evaluationReferenceInputs. You can provide an expectedResponse (used by Correctness), assertions (used by GoalSuccessRate), or an expectedTrajectory (used by trajectory evaluators). Traces without ground truth fall back to ground-truth-free evaluation, so you only need to provide it for the turns you care about. Architecture The pipeline deploys two AgentCore runtimes behind a shared Cognito user pool. One AgentCore runtime for the Strands agent and one for the MCP server: A single Cognito user pool serves two auth flows: The GitHub Actions pipeline deploys the agent stack to a dev environment, retrieves a JWT token from the provisioned Cognito instance to authenticate API calls, invokes the agent with an evaluation dataset, and analyzes the generated traces in Amazon CloudWatch Logs to assess performance against defined thresholds. It automatically approves or blocks the PR based on whether the overall score meets the acceptance criteria. Handling OAuth-protected MCP servers in CI When your agent calls an MCP server protected by OAuth, CI pipelines face a challenge: they don’t have user context. The MCP server expects a JWT with role claims, but a headless CI runner can’t complete an interactive OAuth consent flow. There are three approaches for evaluating the agent, each with different trade-offs: Approach A: Evaluate stored traces Decouple evaluation from live MCP calls entirely. A staging pipeline runs the agent with representative prompts, captures traces, and commits them as JSON fixtures. PR-time, CI evaluates those stored traces. No live invocation is needed. The Evaluate API doesn’t need a live agent. It scores OpenTelemetry spans you provide. Your CI pipeline becomes deterministic (check existing traces) and you sidestep the OAuth problem completely. Trade-off: You’re evaluating the staging deployment’s behavior, not the code in the current PR. The accompanying repo includes scripts/evaluate_stored_traces.py and sample fixtures in fixtures/ to get started with this approach. Approach B: Service account with pre-authorized consent Create a dedicated test user in your identity provider. Complete the OAuth consent flow once (interactively), cache the refresh token in AWS Secrets Manager. CI uses this token to invoke the agent as that test user. Trade-off: Refresh tokens expire. You need a rotation mechanism or periodic manual re-consent. Approach C: M2M auth (this post’s approach) Configure your MCP servers to support both M2M and user-scoped grant types. CI uses M2M tokens while interactive users go through the standard OAuth consent flow. The MCP server middleware distinguishes between token types: M2M tokens contain scopes but no roles, so role checks are bypassed, and all tools are accessible. User tokens carry custom:roles claims, so tool-level access control is enforced. This bypass is secure because M2M tokens require a client secret that’s never exposed to end users. Only CI pipelines and the agent runtime can obtain these tokens, preventing untrusted callers from acquiring role-less tokens. Trade-off: M2M tokens bypass role checks by design. If you need CI to test role enforcement specifically, use Approach B. Use the decision tree below to find out which approach suits your use case: Tip: Start with Approach A to get a quality gate running quickly. Graduate to Approach C (this post) for full end-to-end CI that tests the actual PR’s code changes. Prerequisites - AWS account with AgentCore access and CDK bootstrapped. - Docker installed and running. - Python 3.12+, Node.js 20+. - Install the required Python packages: pip install boto3 requests bedrock-agentcore-starter-toolkit Note: The Evaluation class from bedrock-agentcore-starter-toolkit handles trace collection from CloudWatch and scoring automatically, so you don’t need to manually query log groups or call the raw Evaluate API. MCP server: three-layer auth For Approach C, the MCP server uses three layers to support both M2M and user-scoped tokens. This is the key pattern that makes CI evaluation work alongside production role enforcement. Layer 1 JWT validation (AgentCore): The platform validates signature, issuer, audience, and expiry before the request reaches your code. No implementation needed. AgentCore handles this through the Custom JWT Authorizer. Layer 2 Header passthrough: request_header_allowlist=["Authorization"] on both runtimes makes sure the JWT reaches the agent and MCP containers. AgentCore forwards the caller’s Authorization header to your container unchanged. Layer 3 Role-based tool access (AuthMiddleware): A FastMCP native middleware that reads the JWT through fastmcp.server.dependencies.get_http_headers(), decodes claims using PyJWT, and enforces custom:roles against the tool meta. M2M tokens (scopes but no roles) get full access. User tokens need the right role. The middleware is added directly to the FastMCP server instance: Infrastructure: CDK stack The CDK stack deploys everything in one command: Cognito pool, both runtimes, IAM roles, and pre-created test users. See infrastructure/stack.py for the full implementation. Key resources created by the stack include: - A Cognito domain. - An M2M app client (client_credentials flow) for CI. - A user app client (authorization_code flow) for interactive use. - Two pre-created users: user-a (FinanceUser) anduser-b (HRUser). - An MCP server AgentCore runtime (protocol: MCP) with JWT authorizer. - A Strands agent AgentCore runtime (protocol: HTTP) with JWT authorizer. The GitHub Actions workflow deploys the CDK stack in a dev environment and uses the created resources to run evaluation. Evaluation script The unified evaluation script (scripts/agentcore_eval.py) handles the full pipeline: get token, wait for runtime, invoke agent, wait for traces, run evaluations, and gate on threshold. The token acquisition uses the standard client_credentials grant. Agent invocation uses HTTPS with a Bearer token (not boto3): The script uses bedrock-agentcore-starter-toolkit’s Evaluation class to run evaluations, which handles trace collection from CloudWatch automatically: Evaluation prompts cover the agent’s full tool surface including built-in tools, public MCP tools, and role-gated MCP tools: GitHub Actions workflow The workflow runs on every PR to main that touches agent code, MCP server, infrastructure, or scripts. It deploys the CDK stack, invokes the agent, runs evaluations, posts results as a PR comment, and tears down the stack. See the full workflow for the complete implementation. The key steps are shown below: Warning: Runtimes stay in CREATING for a few minutes after CDK deploy returns, and invoking one before it’s READY fails with 424 Failed Dependency. The workflow polls get_agent_runtime (via the bedrock-agentcore-control client) until both are READY, then warms up the MCP server before evaluating. CI/CD setup Configure the following components to activate automated agent evaluation in your CI/CD pipeline. 1. Create GitHub OIDC provider (one-time) 2. Create IAM role for GitHub Actions Create a role with trust policy for your repo and permissions for CDK, Amazon Bedrock AgentCore, Amazon Cognito, Amazon Elastic Container Registry (Amazon ECR), and Amazon Bedrock. See the accompanying repo README for the full policy. 3. Add GitHub secret Everything else is read from CDK outputs at runtime. Built-in evaluators reference AgentCore provides multiple built-in evaluators organized by what they assess: This post uses four evaluators: GoalSuccessRate, Correctness, ToolSelectionAccuracy, and ToolParameterAccuracy. The tool-call evaluators are particularly relevant for agents with MCP tools. They verify the agent picks the right tool and passes correct parameters. Tip: Start with four to five evaluators for CI. Add trajectory evaluators for tool-calling agents and safety evaluators (Harmfulness, Stereotyping, Refusal) for customer-facing agents. Use code-based evaluators for deterministic checks. Use the full set of periodic deep evaluations. Testing the pipeline: Fail, fix, pass A reliable way to build confidence in a quality gate is to watch it catch a real regression. Deliberate failure: Change the agent’s system prompt to something unhelpful: Push to a feature branch and open a PR. The pipeline runs and you’ll see: Note: ToolSelectionAccuracy might still pass because the agent may select the right tool even with a bad system prompt. The evaluators measure different dimensions independently. Fix and pass: Restore a proper system prompt, push again. The pipeline re-runs: The PR is unblocked. Warning: LLM-as-judge scores have inherent variance. As a result, the same prompt evaluated twice may produce slightly different scores. Set your threshold with some margin below your target reliability to account for this variance. Lessons from testing We ran this pipeline end-to-end. Here are the gotchas. Save yourself the debugging time. - To invoke OAuth-protected AgentCore runtimes, POST directly to the HTTPS endpoint with a Bearer token rather than using the invoke_agent_runtime() method inboto3 . - Trace propagation takes 30-90 seconds. The evaluation script retries every 30 seconds for up to 10 minutes. Don’t query CloudWatch immediately after invocation. - ARM64 images required. AgentCore runtime requires ARM64 containers. GitHub runners are x86_64. Therefore, use QEMU + Docker Buildx for cross-compilation. - Runtime restart after CDK deploy. Invoking a runtime before it’s READY fails with 424 Failed Dependency, so the workflow polls get_agent_runtime until both runtimes are ready (and warms up the MCP server) before invoking. - sessionSpans is the API parameter name. The Evaluate API acceptsevaluationInput: {"sessionSpans": [...]} . Each call must contain spans from a single session only. Mixing sessions causes aValidationException . - Timestamps must be integers. OpenTelemetry nanosecond timestamps like startTimeUnixNano must be JSON integers, not quoted strings. String timestamps cause aValidationException . Adapting for Microsoft Entra ID This post uses Amazon Cognito, but the architecture is identity-provider-agnostic. If your organization uses Microsoft Entra ID (formerly Azure AD), the same pipeline works with targeted changes: - Token endpoint: Change the regional endpoint of Amazon Cognito to https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token . - Discovery URL: Change to https://login.microsoftonline.com/{tenant_id}/v2.0/.well-known/openid-configuration . - Client credentials: Replace Cognito client ID and audience with Entra ID application ID and scope URI. Everything else stays the same. The deploy script’s authorizerConfiguration.customJWTAuthorizer structure is identical, the evaluation script doesn’t touch authentication (it uses IAM through boto3), and the GitHub Actions workflow structure is unchanged. Trade-offs and limitations - Pipeline takes ~10 minutes. CDK deploy + runtime startup + trace propagation + evaluation. Fine for PR gates, too slow for pre-commit. - LLM-as-judge has inherent variance. The same trace evaluated twice may produce slightly different scores. Set thresholds with margin. - Cost per run. Each evaluator invocation calls the judge model. 4 evaluators × 5 prompts = 20 judge calls per PR. Monitor Bedrock costs at scale. - M2M tokens bypass role checks. By design, CI needs access to all tools. If you need CI to test role enforcement, use Approach B (service account). Note: Despite these limitations, automated evaluation is strictly better than no evaluation. Even imperfect quality gates catch obvious regressions that manual review misses. Clean up The accompanying repo provisions two AgentCore runtimes, a Cognito user pool, IAM roles, and pre-created users, so tear everything down when you’re finished to stop incurring cost. A single command removes it all: If you deployed with npx rather than a global CDK CLI, run npx aws-cdk@2 destroy --force instead. In the dev stack, cdk destroy removes both runtimes, the Cognito pool and its app clients, the pre-created users, the IAM roles, and the M2M client secret in AWS Secrets Manager. The secret’s removal policy is set to destroy, so repeated deploys and teardowns stay clean. The CI workflow tears the same stack down automatically at the end of every run, because its CDK destroy step runs with if: always(). You only need this command for stacks you deploy yourself while following along. Key takeaways - Three-layer MCP auth (platform JWT validation to middleware claim extraction to tool-level role checks) cleanly separates concerns and supports both M2M and user-scoped flows without code changes. - CDK deploys everything. One cdk deploy creates the Cognito pool, both runtimes, IAM roles, and pre-created users. Onecdk destroy tears it all down. - The bedrock-agentcore-starter-toolkit simplifies evaluation. The Evaluation class handles trace collection from CloudWatch and scoring, so there’s no need to manually query log groups and call the raw Evaluate API withsessionSpans . - Token forwarding bridges CI and production. The agent inspects the incoming JWT to determine if it’s a user token (forward to MCP for role checks) or M2M token (use shared client, bypass roles). Same code serves both callers. - The Evaluate API is decoupled from the agent runtime. You don’t need a running agent to evaluate traces. This is the key insight that makes Approach A (stored traces) work, and it’s a direct path to a CI quality gate. - Start with four evaluators and expand. GoalSuccessRate, Correctness, ToolSelectionAccuracy, and ToolParameterAccuracy cover the basics. Add safety evaluators for customer-facing agents. - Ground truth and code-based evaluators extend the toolkit. Trajectory evaluators are programmatic and offered at no additional cost, ideal for CI. Code-based evaluators run deterministic Lambda checks alongside LLM-as-judge scoring in the same evaluation call. Try it yourself The accompanying repository includes the complete implementation: CDK infrastructure for Cognito and both runtimes, an agent with MCP client and token forwarding, an MCP server with role-based access control, a unified evaluation script, a GitHub Actions workflow, and two walkthrough scripts. Use scripts/deploy_and_test_rbac.py for deployment and role-based access testing, and scripts/evaluation_pipeline.py for the evaluation pipeline. To extend the project, you can start with Approach A by running python3 scripts/evaluate_stored_traces.py against the bundled fixtures without any deployment. From there, try adding a new role-gated tool to the MCP server (see mcp-server/README.md) or creating a custom evaluator with your own LLM-as-judge prompt. Maybe try adjusting EVAL_THRESHOLD per environment (for example, 0.7 for dev, 0.8 for staging, 0.9 for prod). You can also set up online evaluation for continuous production monitoring or compare on-demand to online evaluation for your use case.