# AWS outlines nine stages for evaluating AI agents from development to production

- Source: MechNext — https://mechnext.eu/en/article/41-aws-outlines-nine-stages-for-evaluating-ai-agents-from-development-to-production
- Published: 2026-10-17
- Updated: 2026-10-17
- Category: Industry News
- Language: en (Markdown in other languages: ?lang=de, ?lang=sk, ?lang=pl, ?lang=zh, ?lang=ru)

> An internal AWS system illustrates how incorrect answers can escape operational monitoring. Its evaluation framework checks tools, agents and orchestration, with results reported by AWS.

AWS has described a nine-stage evaluation framework built on Amazon Bedrock AgentCore, drawing on an internal system serving field sellers and leadership through Amazon Quick. Answers are scoped by territory and role. Performance Insights handles sales intelligence, Customer Insights synthesizes customer information, and Data Insights supports text-to-SQL.

Before launch, evaluations caught $410.08 presented as $410.08 million, mislabeled metrics, an unsupported recurring-revenue estimate and calculations presented as sourced facts. Operational monitoring remained green. These examples explain why AWS evaluates answer content alongside service operation.

### Turning business goals into tests

The stages are Foundation, Experiment, Merge (CI), Gate (CD), Canary, Adversary, Drift, Pulse and Feedback Loop. They cover safety, quality, performance, cost and operational control. Tools, individual agents and multi-agent interactions each have their own golden test set, evaluators and promotion criteria. AWS reports that a pre-production mismatch among data, tool and prompt changes cut pass rate from 78% to 38%.

Foundation translates business outcomes into thresholds, alarms, safety policies, latency targets, per-query budgets and operational SLAs. Any of six failures makes an answer inaccurate: metric swap, wrong period, wrong entity, pure hallucination, wrong symbols or unauthorized calculation. Other measures cover relevance, faithfulness, helpfulness, instruction following and tool selection.

AWS combines built-in AgentCore evaluators, six domain-specific evaluators, DeepEval, AutoEval and deterministic ground-truth checks. LLM judges assess qualities without a single correct answer. The version-controlled golden set draws on production queries, observed failures and synthetic edge cases, covering tools, four agent question types, multi-agent interactions, safety, latency and cost.

### Experiments and release controls

Experiment compares changes through AgentCore On-Demand Evaluation using the same tests and evaluators, assessing quality, latency, cost and token context. Inputs, outputs, retrieved context, scores and reasons are retained in S3 and a data lake. AWS reports that three months of iteration raised pass rate from 93% to a stable 96%. Pure hallucinations fell from 138 to fewer than 52 per 10,000 responses, and unauthorized calculations from 59 to fewer than 44.

Merge automatically runs a 15-case sentinel set on every staging deployment. Safety violations and quality regressions beyond tolerance block deployment; latency and token costs are also assessed. Separate tool, agent and orchestration checks publish scores to CloudWatch. AWS says an entity-resolution check blocked a normalization change that would have returned the wrong customer's data.

Gate evaluates the full golden set. Validated tool responses are frozen for replay, separating agent reasoning from live tool variation. An end-to-end LLM-judged pass covers 200 held-out questions. Candidates are blocked if they regress against their previous run, fail to beat production or exceed the permitted share of low-scoring questions. AWS describes catching a parent-company revenue answer to a subsidiary-scoped question.

### Checks during live operation

Canary uses batch evaluation in 30-minute windows, covering the golden set within 24 hours, plus a daily full run at gate thresholds. One failed window is an operational signal; paging requires three consecutive breaches. AWS reports detecting weaker retrieval grounding in the first window despite normal latency and zero errors, then tracing it to the retrieval tool.

Adversary addresses misuse, while safety checks span all nine stages. AWS says Amazon Bedrock Guardrails synchronously checks inputs and outputs on 100% of traffic before answers reach users. Checks cover request scope, prompt injection, unauthorized sensitive-data disclosure, harmful content, policy violations, jailbreaks and prompt leakage.

Drift continuously evaluates a 10% sample of live traffic. Scores feed a Quality-State Calculator tracking quality, latency and cost. AWS says ticker symbols and shorthand exposed identification gaps; expanding tests and normalizing inputs restored accuracy by the next cycle. After a quality gate, AgentCore Gateway splits a traffic slice between current and candidate agents. Full rollout requires a win or draw on every evaluator dimension.

### Feedback and limits of the results

Pulse combines evaluations with AgentCore Observability. Automatic OpenTelemetry instrumentation sends execution telemetry to CloudWatch after Transaction Search is enabled; evaluator scores use a separate log pipeline. S3 and data-lake outputs support conversational and Amazon Quick dashboards with role-specific views. AWS reports that mean time to detect fell from days or weeks to under four hours.

Feedback Loop uses AgentCore recommendations to revise system prompts and tool descriptions, turning low-scoring traces into regression cases. AWS says replacing hardcoded example values with placeholders reduced hallucinations and raised grounding from 93% to 96%, while prompt iteration fell from days to minutes. Grounding and test pass rate should not be treated as the same metric. The article identifies dataset evaluation as public preview. These are AWS-reported results without independent verification in the supplied evidence; the thresholds and samples describe this implementation, not universal defaults or guarantees.