Amazon has announced GPU partitioning in SageMaker Inference through NVIDIA Multi-Instance GPU, or MIG. The capability divides one physical GPU into hardware-isolated partitions, each with a fixed share of compute and a dedicated block of memory. Depending on the GPU, MIG can create as many as seven instances. Every instance has its own compute resources, cache, memory, and memory bandwidth.

The central deployment choice is a supported partition profile sized for a model rather than an automatic allocation of the entire device. After users select profiles for their models, SageMaker AI handles GPU configuration, placement, and scaling. Hardware isolation between MIG partitions should not be read as a complete tenancy boundary: models placed on the same instance still share the host CPU, system memory, storage, networking, and endpoint routing.

Supported endpoint scope

GPU partitioning works with real-time endpoints that use inference components. It does not support single-model endpoints, serverless endpoints, heterogeneous instance pools, or asynchronous workloads. Teams therefore need to check the endpoint form as well as the accelerator before treating partitioning as an option for an existing deployment.

A partition cannot span accelerators. Models that require a full accelerator or multiple GPUs must continue to use whole devices. The requested profile must be one offered by the selected GPU, while the layout's combined compute and memory units must remain within the capacity of a single device. SageMaker rejects an invalid layout instead of silently falling back to an entire accelerator.

Two optional fields, no new operations

The feature adds two optional API fields. AcceleratorPartitionLayouts on a production variant declares the accelerator layout. AcceleratorPartitionType on an inference component selects the partition type assigned to a model. Partitioning introduces no new API operations, resource types, or IAM actions. If both fields are omitted, existing endpoint behavior is preserved, and the way applications invoke a model remains unchanged.

Partition layouts are chosen when an endpoint is created. Changing the partition profile of an inference component requires a new component. Changing the endpoint's GPU carve requires a new endpoint followed by traffic migration. The selected layout is consequently a deployment-lifecycle decision, not merely a setting that can be switched freely on an existing endpoint.

Results reported by Amazon

Amazon reported a benchmark involving Llama-3.1-8B-Instruct. In that test, one H100 mig-3g.40gb partition delivered more output tokens per second, lower inter-token latency, and a shorter time to first token than one full A100-40GB. The H100 partition led on those measures across all traffic shapes and concurrency levels included in the test.

The same benchmark reported that the H100 partition was 18–34 percent cheaper per million output tokens once workload-dependent crossover points were exceeded. Below those crossover points, the whole A100 was cheaper. These results belong to the tested model, endpoints, configurations, traffic shapes, concurrency levels, prices, and Region. They do not establish the outcome for other workloads, which require case-by-case measurement.

Availability and billing

GPU partitioning is available on the P4d, P5, G7, and G7e instance families in US East (N. Virginia), US East (Ohio), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Tokyo), Europe (Frankfurt), and Europe (Ireland). Instance availability varies by Region, and the announcement does not establish that every supported instance type is offered in every Region on that list.

Customers continue to be billed for the full GPU instance at its existing hourly price. Partitions are not billed independently, and partitioning does not reduce or otherwise change the instance's hourly charge. Any economic result therefore depends on the amount of useful work obtained from the paid instance and on measurements for the particular workload, rather than on a separate per-partition price.

For deployment planning, the relevant sequence is to confirm that the workload uses a supported real-time endpoint, choose a profile that the selected GPU offers, and ensure the total layout remains within one device's capacity. The need to replace components or endpoints when profiles or GPU carves change also makes validation before traffic migration especially relevant. These constraints define where MIG can provide a right-sized allocation while preserving the established invocation interface.