A decision model approaches evaluation differently from a model that generates free-form text. It selects from a fixed set of options and returns calibrated confidence scores or a probability distribution over predefined answers. This structure fits narrow, repeatable evaluation questions because every output has a known interpretation and can be converted into a consistent result.

The AWS post describes how to bring that approach into Amazon Bedrock AgentCore Evaluations. AgentCore Evaluations supports on-demand evaluation during development, batch evaluation of stored sessions, and online evaluation of production traffic. It offers 15 built-in evaluators and also supports custom evaluators defined either as LLM-as-a-judge checks or as code-based evaluators.

A Lambda bridge to decision models

A code-based evaluator runs custom logic in AWS Lambda. That logic can call an external system, model, or API, including a decision model. The implementation discussed in the post uses this extension point to connect two backends: TypeSafe's hosted Jev model and the self-hosted, open-source Strands Decider 2B model.

Jev and Strands Decider support noul yes-or-no questions, choice questions containing as many as 255 options, and score questions with two to ten ordered levels. These formats turn an evaluation criterion into a bounded decision rather than an unconstrained written response.

The HR Assistant example

The example evaluates an HR Assistant built with Strands Agents and deployed to AgentCore runtime. Its evaluation setup combines deterministic checks, built-in LLM-as-a-judge evaluators, and decision-model evaluators backed by either Jev or Strands Decider 2B.

The sample's evaluate.py script deploys two deterministic Lambda evaluators and registers them as code-based evaluators. It then performs an on-demand evaluation across five labeled scenarios with three named built-in evaluators. The script also creates an online configuration that samples 100 percent of live sessions.

Decision-model checks are optional additions. The --with-jev option registers JevGroundedness, JevHelpfulness, and JevGoalCompletion. The --with-decider option registers the same three evaluator roles with Strands Decider 2B as their backend.

From telemetry to an AgentCore result

The code-based evaluator receives OpenTelemetry spans, normalizes them, and formats the relevant material for the decision model. For trace-level evaluation, it sends the targeted turn together with a brief amount of prior history. For session-level evaluation, it sends the full conversation. It then maps the decision model's output back into an AgentCore result.

The mapping depends on the question format. For a noul question, the value is the probability assigned to true, while the label is the most probable answer. For a score question, the evaluator calculates the probability-weighted expected level and scales it to the interval from zero to one. For a choice question, it calculates a weighted average using predefined values associated with the available options.

This conversion preserves more information than a bare winning label. The resulting numeric value reflects the distribution returned by the decision model while still fitting the result structure expected by AgentCore Evaluations.

Validate before gating a release

Connecting an evaluator is only one part of the process. The validation procedure starts with a labeled sample drawn from real traffic. Three independent labelers assess that sample, and majority voting produces the reference labels. The decision-model evaluator and the matching built-in evaluators then run on identical inputs.

The analysis covers agreement, calibration, and the individual cases where evaluators disagree. The check should be repeated whenever the wording of a decision question changes or a model version changes. That repetition treats the question and model version as parts of the evaluated configuration rather than as details that can change without review.

The recommended setup keeps built-in evaluators as the broad baseline, then adds validated decision-model checks and exact deterministic checks. Failed sessions and sessions near a threshold can be reviewed through batch evaluations. Release gates should rely only on built-in evaluators and on decision-model evaluators that have been validated against labeled data.

The result is a layered evaluation arrangement in which each type of check has a defined role. Built-in evaluators provide broad coverage, deterministic checks handle exact conditions, and validated decision-model evaluators answer bounded questions through calibrated probabilities. Online evaluation can observe live sessions, while stored sessions can be revisited in batches when failures or borderline results require closer examination.