Kubernetes Primer for GPU Infrastructure. Before diving into GPU scheduling, it helps to understand the core Kubernetes (K8s) building blocks. Kubernetes is a container orchestration platform that manages workloads across a cluster of machines. A Node is a physical or virtual machine in the cluster. A Pod is the smallest deployable unit — one or more containers that share networking and storage; every GPU workload runs inside a Pod. A Deployment manages a set of identical Pods (e.g., 3 replicas of an inference server) and handles rolling updates. A Service provides a stable network endpoint (IP + DNS name) that routes traffic to the Pods behind it. A Job runs a Pod to completion (e.g., a training run) and a CronJob runs Jobs on a schedule. A DaemonSet ensures one Pod runs on every Node (used for monitoring agents like dcgm-exporter). A Namespace is a virtual partition of the cluster that isolates resources and access — each team or tenant typically gets its own namespace. ConfigMaps store non-sensitive configuration data as key-value pairs that Pods can mount as files or environment variables. kubectl is the CLI tool used to interact with the cluster: kubectl get pods lists Pods, kubectl apply -f manifest.yaml creates resources from a YAML file, kubectl describe node shows node details including GPU allocations. The NVIDIA Device Plugin is a DaemonSet that discovers GPUs on each node and advertises them as schedulable resources (e.g., nvidia.com/gpu: 8), so Pods can request GPUs via resources.limits in their spec.

Scheduling, Taints, and Labels. Kubernetes decides which Node runs each Pod through its scheduler. Labels are key-value tags on resources (e.g., workload=training on a Node) and selectors filter by them — Pods use nodeSelector to target specific Nodes. Taints mark a Node as off-limits (e.g., gpu-reserved=training:NoSchedule), and only Pods with a matching Toleration can schedule there — this prevents non-GPU workloads from landing on expensive GPU nodes. ResourceQuotas cap the total resources a namespace can consume (e.g., max 8 GPUs for team-A). RBAC (Role-Based Access Control) restricts what actions users or service accounts can perform — a tenant's ServiceAccount might only be allowed to manage Pods in their own namespace. NetworkPolicies are firewall rules that control Pod-to-Pod traffic, enforcing tenant network isolation.

Standard Kubernetes scheduling is designed for stateless, independently schedulable Pods — a model that breaks down for distributed AI training where a gang of N Pods must start simultaneously or not at all. If only 7 of 8 required Pods schedule, the training job deadlocks waiting for the 8th while holding GPU resources. Volcano and Kueue are Kubernetes-native batch schedulers that introduce gang scheduling (all-or-nothing Pod group placement), job queuing, preemption policies, and GPU topology-aware placement as first-class primitives, making GPU cluster scheduling viable for multi-tenant AI workloads.

Volcano's core abstraction is the PodGroup — a set of Pods that must be co-scheduled atomically. It supports multiple scheduling algorithms: gang (all-or-nothing), backfill (opportunistic use of idle slots), priority queues, and fair-share allocation across teams. Volcano also integrates with MPI, PyTorch distributed, and TensorFlow through dedicated Job plugins that handle Pod role assignment (master/worker), restart policies, and cleanup. Kueue, Google's newer offering now part of upstream Kubernetes, provides a queue-based model with ClusterQueue (cluster-wide resource pool), LocalQueue (namespace-scoped queue that borrows from a ClusterQueue), and ResourceFlavor (labels that distinguish hardware types like h100-80gb vs a100-40gb) abstractions that map well to multi-tenant GPU pools with heterogeneous hardware.

For AI platform architects, the scheduler choice shapes the entire multi-tenant GPU sharing story. Key capabilities to evaluate include: topology-aware scheduling (co-locating Pods on Nodes connected by NVLink), GPU sharing via MIG (Multi-Instance GPU) partitioning, preemption policies for priority jobs, and integration with autoscaling (KEDA for event-driven scaling, Karpenter for node provisioning). In practice, Kueue is the strategic choice for teams already on modern Kubernetes (1.27+) due to its upstream status and active development; Volcano remains dominant in HPC-oriented deployments and where MPI workloads are primary.

Search Results