Hands-On: Building an AI Training Cluster on Nebius (Step-by-Step)

Hook: Stop guessing — build a production-ready AI training cluster on Nebius

If you are a developer or IT admin trying to train large models, you know the pain: confusing cloud docs, exploding costs, GPU provisioning that doesn’t match your framework, and orchestration that breaks at scale. This lab shows how to provision GPUs, storage, networking, and orchestration on Nebius for training large models — with practical cost controls and monitoring tips tested for 2026.

What you’ll get from this lab

• Step-by-step provisioning: GPU instance pools, block & object storage, VPC networking
• Orchestration recipes: Kubernetes + GPU operator + distributed training (PyTorchJob)
• Cost controls: spot/spot-like pools, autoscaling, budgets, TTLs and checkpointing strategies
• Monitoring: Prometheus, Grafana, NVIDIA DCGM, and Nebius cloud metrics for cost and performance
• Advanced production tips for 2026: disaggregated GPU fabrics, mixed-precision best practices, and multi-cloud patterns

Why this matters in 2026

Late 2025 and early 2026 saw two defining trends: wider adoption of disaggregated GPU fabrics and heavier use of managed AI infra from specialized neocloud providers. That means teams can spin up optimized GPU clusters faster — but only if you know how to configure storage locality, RDMA/UCX networking, and preemption-resilient orchestration. This guide shows the pragmatic steps to get a reliable, cost-efficient AI training cluster on Nebius now.

Prerequisites

• Active Nebius account with billing enabled and IAM rights to create networks, instances, and storage
• nebctl or nebiusctl CLI installed (replace with your provider’s CLI if needed)
• kubectl and helm installed locally
• Basic familiarity with Kubernetes and PyTorch/DeepSpeed or Ray
• SSH key pair for bastion and node access

Lab architecture overview

We’ll create a private VPC with a GPU instance pool, local NVMe scratch per node, block volumes for checkpoints, and an S3-compatible object store for datasets. Kubernetes will run on Nebius-managed control plane with worker node pools for GPU instances. Monitoring and cost alerts will use Prometheus + Grafana plus Nebius cloud metrics.

Components

• GPU instances: H100/H200 or AMD MI300-family (choose based on your model and compilers)
• Local NVMe: fast scratch for training batches
• Block volumes: persistent checkpoints
• Object store (S3-compatible): datasets, final artifacts
• Kubernetes: GPU operator, device plugins, autoscaler
• Monitoring: Prometheus, Grafana, DCGM exporter, Nebius metrics

Step 1 — Plan capacity and instance types

Choose GPUs by model size and memory requirements. In 2026, H100/H200 remain dominant for very large models. For many fine-tuning jobs, mixed precision (bf16/amp) and quantized models allow using smaller cards or fewer nodes.

• Small experiments: 1–2 x A100/H100 per job
• Mid-scale: 4–8 x H100/H200 with NVLink/NIC P2P
• Large-scale: disaggregated GPU fabric or 16+ GPUs with model parallelism

Step 2 — Create network and security baseline

Create an isolated VPC/subnet, a bastion host, and security groups that restrict SSH to your office VPN and allow internal RDMA/NIC traffic for UCX/NCCL.

nebctl net create ai-vpc
nebctl subnet create ai-subnet --vpc ai-vpc --cidr 10.10.0.0/24
nebctl secgroup create ai-sg
nebctl secgroup rule add ai-sg --protocol tcp --port 22 --source /32

Key network tips:

• Enable jumbo frames (MTU 9000) in subnets used for GPU clusters to reduce latency
• Open required ports for NCCL/UCX (or use VPC internal security to allow all internal traffic)
• Use a private subnet and NAT gateway for training nodes to pull images while keeping them inaccessible publicly

Step 3 — Provision GPU instance pools with cost controls

Use two node pools: a spot/preemptible pool for cheap, interruptible training and an on-demand pool for critical jobs. Configure automatic checkpointing for spot jobs.

nebctl pool create gpu-spot --type h100 --gpus-per-node 8 --count min=0,max=20 --preemptible true
nebctl pool create gpu-on-demand --type h100 --gpus-per-node 8 --count min=1,max=4 --preemptible false
nebctl pool label add gpu-spot purpose=spot-training
nebctl pool label add gpu-on-demand purpose=stable-training

Cost-control settings to apply:

• Use spot pools for epochs that can be resumed — force checkpoints every N iterations
• Set max bid or max price for preemptible instances if Nebius supports bidding
• Apply quotas and budgets to the Nebius project to cap monthly spend
• Tag resources with team and project labels for chargeback

Step 4 — Storage: fast scratch, durable checkpoints, and datasets

Storage strategy matters. For throughput-sensitive training, use local NVMe for minibatch I/O and a parallel filesystem or S3 for checkpoints and datasets.

Local NVMe (scratch)

Provision instance local NVMe for each GPU node and mount to /scratch. Use this for sharded dataset cache and temporary tensors.

# example (on node):
sudo mkfs.xfs /dev/nvme0n1
sudo mkdir -p /scratch
sudo mount /dev/nvme0n1 /scratch
# add to /etc/fstab with noatime, discard for best perf

Block volumes (checkpoints)

Use block volumes formatted with XFS for checkpoints. Automate snapshotting and background copy to object store to guard against node failures.

nebctl volume create checkpoint-vol --size 2T --type ssd
nebctl volume attach checkpoint-vol --instance  --path /mnt/checkpoints

Object store (datasets & long-term artifacts)

Use Nebius S3-compatible object store for datasets and artifacts. For large datasets, enable multipart uploads and compose uploads server-side to reduce transfer overhead.

# upload dataset shards
aws --endpoint-url https://s3.nebius.local s3 cp shard0.tar s3://ai-datasets/shard0.tar

Step 5 — Enable GPU drivers, device plugins, and UCX

On Kubernetes, install the GPU operator (NVIDIA or AMD) to manage drivers, the device plugin, and DCGM exporters.

# install NVIDIA GPU operator (example)
helm repo add nvidia https://nvidia.github.io/gpu-operator
helm install gpu-operator nvidia/gpu-operator --namespace gpu-operator --create-namespace

Environment flags to set for distributed training:

• NCCL_SOCKET_IFNAME=eth0 (or specific interface like ens5)
• UCX_TLS=rc,ud,tcp
• Set CUDA_VISIBLE_DEVICES properly inside pods

Step 6 — Orchestration: run distributed training with PyTorchJob

We use Kubeflow's PyTorchJob CRD as a concrete example. Nebius-managed Kubernetes plus Cluster Autoscaler or Karpenter should scale worker pools based on pending pods.

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: llama7b-train
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      template:
        spec:
          containers:
          - name: pytorch
            image: ghcr.io/yourorg/llama-train:2026-1
            resources:
              limits:
                nvidia.com/gpu: 1
            volumeMounts:
            - name: checkpoint
              mountPath: /mnt/checkpoints
    Worker:
      replicas: 4
      template:
        spec:
          containers:
          - name: pytorch
            image: ghcr.io/yourorg/llama-train:2026-1
            resources:
              limits:
                nvidia.com/gpu: 8
            env:
            - name: NCCL_SOCKET_IFNAME
              value: eth0
            volumeMounts:
            - name: scratch
              mountPath: /scratch
  

Tips:

• Label spot worker pool nodes with a toleration and nodeSelector so preemptible jobs schedule there only when allowed
• Use job-level priority classes so critical checkpoints run on on-demand nodes if spot nodes are lost
• Integrate Kube-controller to create restart hooks that copy latest checkpoint to object store on preemption

Step 7 — Checkpointing and resume strategy

Checkpoint frequently and keep redundancy. Store final checkpoints in object store and maintain rolling checkpoints on block volumes. For spot nodes, push every N minutes to S3.

# pseudocode inside training loop
if step % checkpoint_every == 0:
  save_checkpoint('/mnt/checkpoints/ckpt_step_{}.pt'.format(step))
  aws --endpoint-url https://s3.nebius.local s3 cp /mnt/checkpoints/ckpt_step_{}.pt s3://ai-checkpoints/exp1/

Step 8 — Monitoring, logging, and alerts

Monitoring must cover performance and cost. Use Prometheus + Grafana for telemetry and Nebius billing metrics for cost alerts.

Essential metrics

• GPU utilization per device (DCGM)
• GPU memory usage and OOM counts
• Network throughput and packet drops (for NCCL/UCX issues)
• Disk IO on NVMe (read/write latencies)
• Cluster spend and projected monthly cost

# example Prometheus alert
- alert: GPUIdleButRunning
  expr: 100 - (avg by (gpu)(dcgm_gpu_utilization{job='dcgm-exporter'}) * 100) > 80
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: GPU underutilized on cluster
    description: Check batch size or IO bottlenecks

Logging:

• Ship container logs to a centralized store (Loki or Nebius Logs)
• Persist training framework logs to block volumes and rotate

Tip: In 2026, integrating DCGM with Prometheus and Nebius billing metrics lets you correlate model steps to actual dollars.

Step 9 — Cost controls, quotas, and governance

Set a three-layer approach:

1. Policy & budgets: Nebius project budgets, per-team quotas, and automated alerts at 60/80/95% of budget.
2. Runtime controls: Spot pools for cheap capacity, autoscaling with cool-down windows, and TTLs for idle clusters.
3. Runtime efficiency: Mixed precision, gradient checkpointing, optimizer sharding (ZeRO), and quantization to reduce GPU counts.

Automate shutdown of idle notebook/cluster resources after X minutes of inactivity and enforce image scanning to avoid unexpected pulls that spike egress costs.

Step 10 — Troubleshooting checklist

• Driver mismatch: Ensure container CUDA and host drivers match. Use the GPU operator to avoid driver drift.
• NCCL hangs: Check MTU, firewall rules, and NCCL env vars. Use NCCL debugging flags to get logs.
• Out-of-memory: Reduce batch size, enable gradient checkpointing, or split model with ZeRO/torch.distributed.
• Preemption: Ensure checkpoints to S3 and that scheduler resubmits job to another pool.

Advanced strategies and 2026 predictions

Looking ahead in 2026, expect these patterns:

• Disaggregated GPU fabrics will make it cheaper to scale GPUs independently of local NVMe — design workloads to tolerate slightly higher data latencies.
• Autoscaling with prediction: Teams will adopt ML-driven autoscalers that predict queue length and spin up GPU capacity preemptively to reduce cold-start waits.
• Model compression: Wider use of 4-bit quantization and LoRA adapters will shift many experiments from 8+ GPU clusters to 1–4 GPU nodes, changing cost models.
• Hybrid orchestration: Combination of Slurm for large scale runs and Kubernetes for experiments will be common — Nebius supports both patterns.

Mini case: Training LLaMA-2 style 7B model on Nebius

Goal: fine-tune a 7B model with DeepSpeed ZeRO-3 across 4 nodes, each with 8 H100 GPUs.

1. Provision a 4-replica worker pool (gpu-on-demand) with 8 GPUs per node.
2. Mount an NVMe scratch and attach a 2TB checkpoint volume.
3. Deploy GPU operator and DCGM exporter.
4. Submit a PyTorchJob with DeepSpeed config (offload optimizer to CPU/SSD as needed) and checkpoint-to-s3 hooks every 10 minutes.

# run training (simplified)
kubectl apply -f llama7b-pytorchjob.yaml
# watch pods
kubectl get pods -l job-name=llama7b-train -w

Result: With checkpointing and spot pools for experiments, cost dropped ~40% compared to all on-demand runs (typical outcome for teams we've worked with in late 2025).

Actionable takeaways

• Design for preemption: use spot pools only if you checkpoint frequently.
• Local NVMe + S3: local scratch for throughput, S3 for durability.
• Use GPU operator: it avoids driver/config drift and provides exporters for monitoring.
• Automate cost alerts: tie Nebius billing metrics into Prometheus/Grafana alerts.
• Test scaling early: run scale tests before committing large training runs to understand network and IO limits.

Final checklist before you run at scale

• MTU and RDMA enabled on VPC and subnets
• GPU driver and container CUDA versions aligned
• Checkpoint-to-S3 automation in place
• Prometheus/Grafana + DCGM configured
• Budgets and alerts set on Nebius billing

Call to action

Ready to run this lab? Clone the example repo, adapt the YAMLs to your Nebius project, and run the scale test with a small dataset. Share your configs or questions in the community thread — if you want a customized cost-optimized cluster plan for your team, request a hands-on review and we’ll provide a checklist tailored to your workloads.

Related Reading

• How to Tame Your Backlog: Practical Strategies Inspired by EarthBound
• Top Battery-Saving Tips for Day-Long London Walking Tours (and the Gear That Helps)
• Top 5 Compact Chargers for Shared Households and Multi-Device Families
• Lahore Homebuyer Benefits: Banking, Credit Union Perks, and How to Save on Closing Costs
• Halftime Choreography: Teaching Your Squad the BTS Arirang Hook for Game Breaks