07. Heterogeneous Computing

Chapter focus: The "execution layer" of heterogeneous computing. Upper-layer parallelism partitions the model across GPUs; this chapter makes those GPUs actually run — environment (driver/CUDA), resources (scheduling/queues), and strategy (mixing/selection). Miss any one and nothing runs.

Articles

1. GPU Driver & CUDA Installation Guide — Driver and CUDA Toolkit installation SOP for Linux (CentOS/Ubuntu/SUSE), including SXM Fabric Manager configuration.
2. Slurm Heterogeneous Scheduling: Resource Management for CPU/GPU Mixed Clusters — gres resource declaration, partitions and QOS fairness, heterogeneous GPU classification (Features), and mixed-cluster scheduling pitfalls.
3. GPU Resource Partitioning in Practice: MIG, vGPU and Multi-Tenant Sharing — Slicing one card among multiple jobs/tenants: MIG hardware isolation, vGPU virtualization, soft-partitioning selection and Slurm integration, with isolation-boundary and communication-path pitfalls.

📌 Planned: heterogeneous mixing selection (training + inference cards in one cluster), Triton operator development, heterogeneous cluster monitoring & job-efficiency analysis.

Reading Path

• Setting up environments: start with the CUDA installation SOP for a clean baseline.
• Managing clusters: go straight to Slurm heterogeneous scheduling — 80% of real-world heterogeneity problems live in scheduling, not hardware.
• Performance work: combine with the training chapter (distributed parallelism) to understand GPU partitioning, then return here for the scheduling constraints.

Key Takeaways

• Three scheduling principles for heterogeneous clusters: partitions isolate resources, QOS controls fairness, backfill guarantees utilization.
• gres is Slurm's "common language": a mismatched declaration means jobs PENDING forever; a correct one scales elastically from one GPU to a full node.
• For mixed GPU types, partition by node, keep each node homogeneous — the system-level driver/FM constraints make this the most trouble-free structure.