
SageMaker HyperPod Inference supports distributed serving with NVIDIA Dynamo
Amazon SageMaker HyperPod Inference supports NVIDIA Dynamo, an open-source framework for distributed inference. The integration addresses deployments in which inference work can span multiple GPUs and nodes. Dynamo works with vLLM, SGLang, and TensorRT-LLM and supports multi-node deployment, disaggregated prefill and decode, and routing that takes KV cache placement into account.
Aggregated and disaggregated serving
In a disaggregated topology, the Dynamo frontend routes requests to separate prefill and decode worker pools. Both pools load the model’s full weights, while KV cache data moves between them over EFA. The two processing phases can therefore be assigned to distinct groups of workers, with the cache transfer connecting their work.
Dynamo also supports aggregated serving. In that topology, prefill and decode execute in the same worker. Workers in either topology can use tensor parallelism and pipeline parallelism across one or multiple instances. The architecture can consequently vary along several dimensions: the two phases may remain together or be separated, and an individual worker may extend beyond one instance.
Operators and deployment resources
The HyperPod Inference add-on installs Dynamo custom resource definitions. Responsibilities are then divided between two operators. The HyperPod Inference Operator manages endpoint deployment through InferenceEndpointConfig resources, while the Dynamo operator manages DynamoGraphDeployment resources.
The add-on exposes two deployment paths. A user can bring a DynamoGraphDeployment, or define an InferenceEndpointConfig containing dynamoSpec. Both paths support single-instance and multi-node deployments. With the second path, the supported inference engines are vLLM, SGLang, and TensorRT-LLM; vLLM is the default. This path also exposes an OpenAI-compatible interface.
Scheduling and infrastructure management
A multi-node worker depends on a set of pods being available together. KAI Scheduler applies gang scheduling so that all pods required by such a worker are placed together on GPU resources. This connects the logical multi-node worker definition with coordinated placement of its constituent pods.
HyperPod monitors node, GPU, and EFA health. It manages node replacement and uses its orchestration components to recreate affected workers. It also manages model weights, scales clusters using GPU and inference metrics, monitors node health, and replaces failed nodes. These are described management mechanisms, not a guarantee that every possible deployment or failure condition will recover automatically.
Two configuration-specific benchmarks
One AIPerf benchmark used Qwen3-235B-A22B-FP8 on two p5en.48xlarge instances equipped with 16 H200 GPUs. The highest qualifying disaggregated result was 207 output tokens per second per GPU, compared with 101 for aggregated serving. Within that measured setup, the reported difference was 2.05 times.
A separate AIPerf benchmark used Qwen3.5-397B-A17B-FP8 with eight active H200 GPUs at concurrency 32. Its two-node configuration produced 126.4 output tokens per second per GPU and retained 99.1 percent of the measured single-node throughput.
Both results must remain attached to their stated workloads and configurations. They do not establish performance for other models, hardware, traffic patterns, concurrency levels, topologies, or deployment settings. The figures are useful as specific observations of the tested arrangements, rather than as generalized performance promises.
What teams must choose
The integration does not prescribe one serving layout. A deployment can use aggregated workers or separate prefill and decode pools, operate on one instance or multiple nodes, and select among the three supported engines. A disaggregated design also carries explicit structural details: both worker pools hold full model weights, and KV cache transfers use EFA.
The deployment-resource choice is another part of the design. Bringing a DynamoGraphDeployment gives one path, while describing Dynamo through dynamoSpec in an InferenceEndpointConfig gives another. In multi-node cases, gang scheduling and the selected tensor or pipeline parallelism shape how the worker maps to GPU resources.
Viewed together, the capabilities provide building blocks for distributed inference: endpoint and graph resources, coordinated GPU placement, alternative serving topologies, supported inference engines, health monitoring, model-weight management, and metric-based cluster scaling. The appropriate combination remains dependent on the model and the concrete deployment configuration, and the two published benchmark results should be interpreted only within their measured boundaries.