# How AWS demonstrated distilling a shopping agent into a 4B model

- Source: MechNext — https://mechnext.eu/ru/article/60-how-aws-demonstrated-distilling-a-shopping-agent-into-a-4b-model
- Published: 2026-11-03
- Updated: 2026-11-03
- Category: Industry News
- Language: ru (Markdown in other languages: ?lang=en, ?lang=de, ?lang=sk, ?lang=pl, ?lang=zh)

> An AWS walkthrough transferred agent behavior from GLM-5.3 to Qwen3.5-4B through sequence-level distillation. The student scored 0.541 in one ShoppingBench run, while its IFEval scores declined.

An AWS blog walkthrough describes sequence-level knowledge distillation, in which a teacher model generates responses and a smaller student learns from those responses through supervised fine-tuning. The method requires the teacher’s text, but not its weights, logits, or gradients. In the walkthrough, GLM-5.3 on Amazon Bedrock served as the teacher and Qwen3.5-4B as the student. The student was adapted with serverless supervised fine-tuning through SageMaker AI model customization.

### From shopping problems to successful trajectories

The data came from ShoppingBench, which supplied 750 synthesized shopping problems. The collection contained 250 problems for each of three intent categories: product, shop, and voucher. The walkthrough assigned 426 problems to training and reserved 324 non-overlapping problems for evaluation. This separation matters when reading the reported result because the evaluation tasks were distinct from the problems used to produce the training examples.

For every training problem, the teacher was sampled twice at temperature 1.0, producing 850 trajectories. ShoppingBench’s scorer then retained only trajectories that solved the task. Every retained trajectory included the shopper request, the teacher’s reasoning and tool calls, and the observations returned by the environment. The student therefore received a record of a successful multi-step interaction rather than merely an isolated final answer.

### Encoding and validation design

The training encoding produced one row for each trajectory step. Loss was applied only to the assistant turn, allowing the request and preceding interaction to provide context without making every contextual element a prediction target. After the successful trajectories were filtered and expanded by step, the resulting data contained about 2,600 training rows and 130 validation rows.

The training and validation split was performed by whole problem instead of by row. That choice prevented nearly identical contexts from a single trajectory from appearing in both sets. A row-level split could otherwise place different steps of the same underlying problem on opposite sides, weakening the validation set as a check on how the student handles separate problems.

The fine-tuning run used a maximum sequence length of 16,384 tokens, a LoRA rank of 64, and three training epochs. These settings document the particular run in the walkthrough. They do not establish that the same configuration is appropriate for a different task, model, dataset, or workload.

### What the held-out evaluation reported

Evaluation used Absolute Success Rate across all 324 held-out problems. The score was averaged over product, shop, and voucher intents, and missing trajectories counted as failures. In the reported single run, the distilled 4B student reached an average Absolute Success Rate of 0.541. The sampled teacher scored 0.511, while the greedy teacher scored 0.540.

Within this evaluation, the student’s reported score was close to the greedy teacher and slightly above both listed teacher scores. The comparison must remain confined to the described ShoppingBench setup. It does not show that the student generally matches or surpasses frontier models. The source reports one LoRA-student run and says that a few points of run-to-run variation should be expected.

### Why the reasoning trace mattered

An ablation removed the teacher’s reasoning and trained with tool calls alone. The student’s average Absolute Success Rate fell from 0.541 to 0.415. Within this experiment, successful tool calls by themselves did not provide the same training signal as trajectories that also included the teacher’s reasoning. The result supports the walkthrough’s decision to preserve the fuller interaction record, including reasoning, tool use, and returned observations.

### The specialization came with a tradeoff

IFEval exposed a decline in general instruction following relative to the base model. The distilled student’s prompt-strict score decreased from 0.819 to 0.741, while its instruction-strict score decreased from 0.872 to 0.813. The task-specific ShoppingBench result should therefore be considered alongside this measured regression rather than treated as an unqualified improvement.

Taken together, the walkthrough presents a concrete pipeline: sample the teacher more than once, filter for successful solutions, retain the reasoning and environment interaction, split by complete problems, and measure both the target capability and possible regressions. It does not provide workload-independent cost or latency figures, nor measured throughput, savings, or return on investment. Its numerical conclusions belong to the specified models, data split, configuration, and single reported run.