An AWS guide demonstrates multi-turn reinforcement learning, or MTRL, on Amazon SageMaker AI using Wordle. The policy model learns from rewards for multi-step episodes. The authors report improved GPT-OSS-20B solve rates on held-out words, alongside practical lessons about rewards and runtime failures. The results and costs describe this experiment and do not establish performance on other tasks.

How the model and agent work together

Wordle gives the agent six attempts to identify a hidden five-letter word. SageMaker AI manages training and the policy model; Amazon Bedrock AgentCore hosts a Strands agent, the environment and reward function. The agent calls the SageMaker Job Runtime's OpenAI-compatible endpoint and receives feedback through guess_word. GRPO compares rewards across rollouts of the same word to update a LoRA adapter.

The example returns one reward per episode, while MTRL also accepts per-turn rewards through UpdateReward. Agents can run on AgentCore or customer infrastructure such as EKS, EC2 or Fargate, reached through an AWS Lambda forwarder. The guide describes the service as serverless, with training and sampling infrastructure provisioned by the service.

Tasks without target answers

Dataset rows contain tasks without target responses, transcripts or labels. The service passes the prompt column, or the first column, verbatim to the agent. The example includes an instruction, secret answer and tracing identifier, but the policy model never sees the answer. It uses 640 training words and 128 nonoverlapping validation words. AWS states that datasets, agents and trained models remain in the customer's account and are not used to train base models.

The launcher uses MultiTurnRLTrainer and requires an MLflow app. It sets a sampling limit of 8,192 tokens, a maximum of 20 epochs and 100 steps. With batches of 128 tasks and eight rollouts per task, an epoch has five steps and each step plays 1,024 games. The guide warns that redeploying the agent during training restarts the runtime and can fail the entire job.

Rewards must preserve the outcome ordering

The initial reward function accumulated bonuses for individual guesses. A loss could earn approximately +0.49, and a third-guess solve could outscore a first-guess solve. The revised function combines a dominant outcome term with bounded shaping. The shaping component is clipped to between −1 and 1, with weights of 0.04 for solves and 0.15 for losses.

According to the authors, this preserves the ordering of faster solves, slower solves, losses and games with no valid guesses. Even the lowest-scoring solve earns +1.01, above the service's success threshold of 1.0. The reward therefore needs to express the goal of the complete game as well as any intermediate progress.

Evaluation settings changed the results

At medium reasoning effort and temperature 1.0, the authors report a solve-rate increase from 61 to 84 percent. The 84 percent figure averages two runs scoring 80 and 88 percent. At low effort, the rate rose from 15 to 69 percent. Training validation at temperature 0, however, peaked at 48 percent and ended at 37 percent. These figures describe different evaluation conditions.

The authors did not test whether additional training steps improve results at serving settings. They report that low reasoning effort used 63–83 percent fewer sample tokens per evaluation game than medium effort. MLflow records rewards, tokens and turn statistics, while CloudWatch OpenTelemetry traces capture model turns and tool calls.

Operational lessons and reported costs

At 32 concurrent rollouts, the authors encountered credential-provider starvation; hour-long token caching also caused authentication failures. Their fix uses a self-refreshing session, a short cache and re-signing after a 403 response. Games with no valid guesses receive −1.5; there were 51 in the 100-step run. Excessive learning rates also turned tool calls into plain text, prompting guess recovery with a formatting penalty.

The 100-step run completed 105,486 rollouts in 12 hours and 40 minutes, consuming 491.6 million prefill, 159.7 million sample and 643.3 million training tokens. The authors reported approximately $339 for training tokens and $79 for AgentCore runtime at then-current us-east-1 list prices. The evaluation series cost about $1.20 in tokens and took around an hour; individual 128-word evaluations took four to seven minutes.

Training and evaluation jobs stop billing when complete, but the AgentCore runtime, its IAM role, MLflow app and S3 objects persist. The guide states that the training role and model packages are not billed. These findings come from the AWS walkthrough, rather than independent verification, and its reported prices are not a current cost estimate.