Building GPU Platforms on Kubernetes

Build the platform other teams depend on

Your organization has GPU infrastructure. Several teams need access, with different workloads, permissions, and performance requirements. You own the platform decisions that connect them.

• Make GPUs available to Kubernetes workloads and diagnose failures across drivers, container runtimes, and device allocation.
• Share GPUs between workloads with time-slicing, MPS, or MIG. Determine which GPU memory, compute, and fault-isolation guarantees each approach provides.
• Find wasted GPU capacity by connecting GPU allocation and DCGM metrics to application throughput, memory use, and scheduling constraints.
• Provision and manage GPU-backed Kubernetes clusters with NVKE, from workload-cluster creation to node-pool changes and node replacement.

1. Day 1

   1. 1. Linux containers and Kubernetes

      Lecture + hands-on labs + challenges

      Kubernetes doesn't know how to deploy apps written in languages such as Java, Node.js, or .NET. Instead, it only understands and uses Linux containers. But how do these Linux containers work? Are those strictly necessary to master Kubernetes? In this course, you will familiarise yourself with containers and Docker.

      1. Containers vs VMs

      2. Understanding process isolation

      3. Is Docker the only one?

      4. Docker client-server architecture

      5. Running containers

      6. Building Docker images

      7. Mounting volumes

      8. Exposing ports

      9. Managing containers lifecycle

      10. Injecting environment variables

      11. Debugging running containers

   2. 2. Kubernetes fundamentals

      Lecture + hands-on labs + challenges

      In this course, you will learn how to build, deploy and scale your application in a Kubernetes cluster. You will also learn how to expose your app to the internet and architect apps that are designed to be horizontally scalable.

      1. Managing containers at scale

      2. The battle of container orchestrators

      3. Visualising the data centre as a single VM

      4. The scheduler is the best Tetris player

      5. Kubernetes as an infrastructure as a service provider

      6. What are Pods, Services, and Ingresses?

      7. Creating a local cluster with minikube

      8. Creating a Deployment

      9. Exposing Deployments

      10. What is a Pod?

      11. Scaling applications

      12. Testing resiliency

   3. 3. Deployment strategies

      Lecture + hands-on labs + challenges

      Every time you deploy new features in production, you shouldn't stop your app, change the version and start it again. With zero-downtime deployments, no one notices any glitch while you're upgrading your apps. How does that work in Kubernetes? In this course, you will learn how Services work in Kubernetes and how you can roll out changes with zero downtime using rolling updates. You will also learn about Canary Deployments and Blue-green deployments.

      1. Monitoring for uptime

      2. Liveness probe

      3. Readiness probe

      4. Startup probe

      5. Executing zero downtime deployments with rolling updates

      6. Using labels and selectors

      7. Testing features with canary deployments

      8. Releasing features with blue-green deployments

      9. Understanding rollbacks and ReplicaSets

2. Day 2

   1. 1. Templating Kubernetes resources

      Lecture + hands-on labs

      Resources in Kubernetes are described as YAML files. If you wish to have the same resources for different environments such as development, preproduction and production, you may be tempted to copy the files three times. Or you could use a templating engine. In this course, you will learn how to do that with Helm, Kustomize or with your programs in any language.

      1. Creating reusable templates

      2. Helm's templating engine

      3. Understanding the Helm architecture

      4. Templating resources with Go and Sprig

      5. Managing releases with Helm

      6. Reverting changes with rollbacks

      7. Depending on other charts

      8. Storing reusable templates in repositories

      9. Structured templating introduction with YQ

      10. Kubernetes specific structured templating with Kustomize

      11. Using Kubernetes SDKs for templating

   2. 2. Kubernetes Architecture

      Lecture + hands-on labs

      Can Kubernetes recover from failures? In this course, you will learn the Kubernetes core components by building a cluster from scratch using kubeadm. But the best way to learn something is by breaking it. So you will also break the cluster one node at the time and observe what happens.

      1. Single and multi-node clusters

      2. Examining the control plane

      3. Persisting changes in etcd

      4. Syncing changes with RAFT

      5. Event-based architecture

      6. Understanding the kubelet

      7. Verifying "no single point of failure"

      8. Setting up a multi-master cluster

      9. Investigating multi-master setup in EKS

      10. Exploring multi-master setup in Monzo

      11. Creating a 3 node cluster with kubeadm

      12. Installing an overlay network

      13. Installing an Ingress controller

      14. Exploring the API without kubectl

      15. Taking down the cluster one node at a time

3. Day 3

   1. 1. Cluster networking and east-west traffic

      Lecture + hands-on labs

      How does Kubernetes handle internal traffic between your microservices? How can two apps in the cluster talk to each other? How is Kubernetes assigning IP addresses to Pods and Services? In this course, you will explore how the traffic is routed inside the cluster.

      1. Network challenge with multiple nodes

      2. Network routing recap

      3. Understanding network requirements

      4. Understanding cluster address limits

      5. Assigning IP addresses

      6. The challenge of service discovery

      7. Exploring the Endpoints

      8. Balancing in-cluster traffic

      9. Routing traffic with kube-proxy

      10. Discovering Service types: headless and ClusterIP

      11. Network security for east-west traffic

      12. Network policy targets

      13. Network policy best practices

   2. 2. North-south traffic and Service Meshes

      Lecture + hands-on labs

      How to expose application frontends to the internet from a Kubernetes cluster? What's the solution for Layer 4 and Layer 7? In this course, you will learn about more Service types, and the Ingress.

      1. Layer 4 solution to expose a service

      2. The details of using Nodeport

      3. Provision loadbalancers automatically

      4. Helping out the world to save on IPs

      5. HTTP based routing for domains and paths

      6. Routing traffic with Ingress manifests

      7. The battle of Ingress controllers

      8. End-to-end packet journey

      9. Service mesh use-cases

      10. The ins and outs of service meshes

   3. 3. Advanced scheduling

      Lecture

      Master advanced placements of workloads in your infrastructure. Learn how to schedule machine learning deployments to nodes with GPU or how you can segregate workloads for regions (useful if you need to comply with policies and regulations).

      1. How the Scheduler works in Kubernetes

      2. Filters and predicates

      3. nodeSelector

      4. Node affinity

      5. Pod affinity/anti-affinity

      6. Taints and tolerations

      7. Topology Spread Constraints

      8. Custom schedulers

4. Day 4

   1. 1. NVIDIA GPU allocation, sharing, and isolation

      Lecture + hands-on labs

      Make NVIDIA GPUs available to Kubernetes workloads. Compare GPU sharing mechanisms and the memory, compute, and fault-isolation guarantees they provide.

      1. NVIDIA drivers, Container Toolkit, and GPU Operator

      2. Device plugins and Dynamic Resource Allocation (DRA)

      3. GPU requests and device access inside a container

      4. GPU time-slicing, Multi-Process Service (MPS), and Multi-Instance GPU (MIG)

      5. GPU memory limits, compute sharing, and tenant isolation

      6. Diagnosing GPU access failures and competing workloads

   2. 2. NVIDIA GPU monitoring and resource optimization

      Lecture + hands-on labs

      Find wasted GPU capacity by comparing Kubernetes allocation, NVIDIA DCGM metrics, and application throughput. Investigate the constraints that prevent workloads from using available GPUs.

      1. GPU allocation versus hardware activity and useful work

      2. NVIDIA DCGM and dcgm-exporter

      3. SM activity, memory use, and application throughput

      4. CPU, memory, and placement constraints on GPU nodes

      5. Multi-GPU fragmentation and stranded capacity

      6. Assessing a resource change against throughput and latency

   3. 3. Cluster lifecycle with NVIDIA Kubernetes Engine (NVKE)

      Lecture + hands-on labs

      Explore NVKE through its local evaluation environment. Connect workload-cluster provisioning and node-pool changes to the GPU runtime and the teams that use the platform.

      1. NVKE management and workload clusters

      2. NICo infrastructure and NVKR build artifacts

      3. Creating and accessing a workload cluster

      4. Node pools, configuration changes, and node replacement

      5. Cluster separation and GPU runtime responsibilities

      6. Cluster health, telemetry, and preview capabilities

Before you join