# AWS reports cross-Region HyperPod training results with Qumulo

- Source: MechNext — https://mechnext.eu/en/article/22-aws-reports-cross-region-hyperpod-training-results-with-qumulo
- Published: 2026-09-27
- Updated: 2026-09-27
- Category: Industry News
- Language: en (Markdown in other languages: ?lang=de, ?lang=sk, ?lang=pl, ?lang=zh, ?lang=ru)

> AWS tested a HyperPod cluster in Oregon reading data held in Ohio. Its post reports an initial cache warmup, followed by a steady training rate of 115–117 samples per second.

On September 25, 2026, AWS published [Multi-Region training with Amazon SageMaker HyperPod and Qumulo](https://aws.amazon.com/blogs/machine-learning/multi-region-training-with-amazon-sagemaker-hyperpod-and-qumulo). The post describes a training validation in which compute in one AWS Region accessed data held in another. The architecture and measurements below are AWS's reported results from that validation.

### Where the data and compute sat

The hub cluster was in US East (Ohio), alongside the data. A spoke cluster in US West (Oregon) read the data remotely through Qumulo Cloud Data Fabric. AWS reported 60 ms of network latency on the cross-Region connection. That figure describes the connection between the Regions; it should be kept distinct from the read latencies reported for individual phases of training.

Each HyperPod cluster mounted a local Cloud Native Qumulo instance over NFS. The spoke's instance retrieved data from the hub over VPC peering. This arrangement matters when interpreting the results: the spoke's training compute was remote from the data, while its NFS mount connected to a local Qumulo instance. The post therefore describes both the cross-Region data path and the local interface used by the training cluster.

### The workload behind the numbers

The reported job trained a 1.02 billion-parameter LLaMA v3 model. Each cluster used two ml.p5.48xlarge instances, totaling 16 H100 GPUs per cluster. AWS used PyTorch 2.1 FSDP, a batch size of 24 per GPU, and a sequence length of 4,096 tokens. The walkthrough used the C4 dataset tokenized for LLaMA, with 48.4 million sequences packed into 4,096-token windows.

These details define the conditions under which the training rates were observed. Samples per second is useful here as a measure of this particular job. It does not, by itself, establish how another dataset, access pattern, network connection, or deployment would perform. The reported batch phases also make clear why a single average for the spoke would hide the behavior at startup.

### What happened during training

For the hub run, AWS reported sustained read throughput of 1.0–1.3 GBps, read latency of 2–3 ms, and GPU utilization of 98–100 percent. Those measurements describe training beside the data and provide context for the spoke measurements.

The spoke had a slower initial phase. AWS reported 95–105 samples per second during its first 100–150 batches, then a convergence to 115–117 samples per second. For batches 0–100 of the cold start, the post gave GPU utilization of 80–90 percent and throughput of 500–800 MBps. For batches 100–150, it reported 5–10 ms read latency, approximately 115 samples per second, and 98–100 percent GPU utilization. The phase ranges should be read as a progression through warmup rather than as one fixed rate for every early batch.

After NeuralCache warmed, AWS said 94–96 percent of reads were served from local NVMe cache at sub-5 ms latency. This is the post's explanation for why the remote cluster's observed behavior changed after startup. Cache hit share, read latency, GPU utilization, and training speed are separate measurements; together they describe the reported run without making any one figure a guarantee for a different run.

### How to use the result

The validation provides a concrete example of a hub in Ohio and a spoke in Oregon under the stated model, dataset, hardware, and software settings. Its central observation is the spoke's movement from a slower cold start to the reported steady training rate after the cache warmed. Readers comparing it with their own plans need to preserve those conditions and the distinction between the 60 ms cross-Region network latency and the later local-cache read latency. The source is one AWS account of its test, so the figures are best understood as reported measurements for this configuration.