# Amazon adds deeper checks and workload recovery controls to SageMaker HyperPod

- Source: MechNext — https://mechnext.eu/ru/article/75-amazon-adds-deeper-checks-and-workload-recovery-controls-to-sagemaker-hyperpod
- Published: 2026-11-19
- Updated: 2026-11-19
- Category: Industry News
- Language: ru (Markdown in other languages: ?lang=en, ?lang=de, ?lang=sk, ?lang=pl, ?lang=zh)

> Three HyperPod capabilities address deep training validation on EKS, Kubernetes resource monitoring, and hung-job detection for Slurm clusters.

Amazon introduced three capabilities for Amazon SageMaker HyperPod: **MultiNodeTrainingValidation**, the HyperPod Kubernetes Resource Monitoring Agent, and hung-job detection for Slurm clusters. They operate at different layers. The first validates instances with a fixed training workload, the second observes a Kubernetes environment, and the third identifies participating Slurm jobs that have stopped making progress.

### Deep training validation for EKS clusters

MultiNodeTrainingValidation is a deep health check for HyperPod clusters orchestrated by Amazon EKS. It trains a fixed workload on every instance and then across instances of the same type. This goes beyond treating general node status as the entire test: the instances must execute a defined reference training path.

The validation contains two tests. Tiny Megatron runs on all GPUs of each instance. A Megatron test then runs across instances of the same type over EFA. Either test fails an instance when loss becomes NaN or infinite, or when training exceeds its time limit. The two stages therefore exercise both the GPUs within an individual instance and training across matching instances.

Operators can invoke the validation during instance creation or updates through `OnStartDeepHealthChecks`. They can also apply it to existing instances with the `StartClusterHealthCheck` API. This supports checks tied to infrastructure changes as well as checks initiated separately on resources that already exist.

> A successful result means that the instances correctly trained the fixed reference workload. It does not guarantee that a user’s own training job will converge.

That boundary matters when reading the outcome. The validation provides evidence about the tested instances and reference workload, but it does not replace evaluation of a user’s model, data, or training behavior.

### A reporting agent for Kubernetes resources

The HyperPod Kubernetes Resource Monitoring Agent runs as one pod per cluster in the `aws-hyperpod` namespace and prefers CPU nodes. Its role is observational: it reports problems but does not reboot nodes or restart jobs. Remediation therefore remains separate from the agent’s findings.

The agent watches nodes, critical infrastructure pods, DaemonSets, supported training jobs, and opt-in NCCL communication. It reports findings through Kubernetes events and structured CloudWatch logs. These two outputs expose the observations through the cluster’s event stream and through structured logging without making the monitor itself an automatic restart mechanism.

#### Opt-in NCCL communication monitoring

NCCL monitoring requires NCCL 2.24 or later and must be enabled for the relevant job. For an opted-in job, the agent reads the NCCL RAS report from every pod every 30 seconds. It combines those pod-level results into one job-level state: healthy, degraded, or unhealthy. The summary thus describes the job as a whole while still being based on reports gathered from every participating pod.

### Hung-job detection for Slurm

For HyperPod Slurm clusters, hung-job detection runs as an agent on every node. It is disabled by default and monitors only jobs whose `HYPERPOD_HANG_RULES` environment variable points to a rules file. A job is therefore not covered merely because it runs in the cluster; it must be configured to participate.

The agent can use default detection based on job output and GPU activity. Alternatively, operators can supply custom regular-expression rules describing progress and errors. Custom rules replace the default detection for that job, so the two approaches are alternatives rather than cumulative rule sets.

When the agent detects a hang, it writes a `HANG_DETECTED` event to CloudWatch. With the default `notify` action, it leaves the job running. With the `cancel` action, it also cancels the job so that Slurm can requeue it. This separates detection and reporting from the optional recovery action.

Hung-job detection requires a HyperPod Slurm AMI released on or after November 10, 2026. Clusters created on or after that date already include it. For a particular workload, the relevant conditions are therefore the cluster image, whether the feature is enabled, whether the job points to a rules file, which detection method is selected, and whether the configured response is notification or cancellation.
