We offer private, instructor-led training for data platform engineers who operate Airflow with VM-based workers and need task execution on Kubernetes.
Your team knows DAGs and the current worker environment. Short-lived task Pods expose dependencies that persistent workers can hide, including installed libraries, credentials, local files, and retained logs.
LearnKube connects those requirements through a representative DAG and selected task. Proposed Airflow exercises explain executor choices and the Kubernetes behavior behind execution and recovery.
Hands-on learning and the skills engineers take back to work.
Preview: course-wide figures are not yet available.
Airflow remains responsible for DAG dependencies when workers move from VMs. KubernetesExecutor runs task instances in worker Pods, while KubernetesPodOperator launches Pods for selected tasks even with another executor. These choices differ in scope. Images, DAG delivery, credentials, and durable logs need explicit configuration when task environments no longer persist.
I separate where Airflow decides a task is eligible from where that task executes. A Pod that exits successfully is only one part of a workflow whose retries, outputs, and logs must remain understandable.
— Daniele Polencic, LearnKube founder and Kubernetes instructor
| Current task or requirement | New concept or decision | Practice |
|---|---|---|
| Assign work to workers | Executor or Pod-based operator | Trace a selected task |
| Install task dependencies | Images and DAG delivery | Diagnose a missing package |
| Retain execution evidence | Durable logs and outputs | Retrieve logs after cleanup |
| Supply task capacity | Pod resources and cluster capacity | Inspect a queued task |
We propose a Kubernetes workshop customized around an existing DAG and one representative task. Your team compares execution options before deciding which worker behaviors need to change.
The exercises keep Airflow's orchestration responsibilities visible. They focus on task images, Kubernetes resources, and operational evidence, rather than assume that moving workers also replaces the scheduler or metadata database.
This proposed agenda connects Kubernetes fundamentals to Airflow task execution. Airflow-specific examples adapt the core course to your chosen executor and operator model.
Day 1
We compare persistent workers with container images and Pod lifecycle. You will review Kubernetes controllers and probes, then distinguish service deployment from the execution of an Airflow task.
Hands-on exercises
Day 2
You will use Helm and compare Kustomize for supporting Kubernetes resources and configuration. We explain controllers, kubelets, and node failures alongside the task state that Airflow records.
Hands-on exercises
Day 3
We trace DNS, Services, and access to data systems, with ingress for supporting interfaces. You will examine network policies, service mesh considerations, resource requests, and placement rules.
Hands-on exercises
Day 4
You will examine durable task logs and outputs, secrets, authentication, and RBAC. We distinguish task concurrency and node capacity from HPA-managed services to clarify which component controls execution demand.
Hands-on exercises
Your DAGs, execution approach, and worker-support responsibilities can shape the agenda. Get in touch to tailor the workshop to your Airflow worker transition.
When a task fails on a new Pod despite working on its old VM, the instructor can compare images, files, and credentials. Your team can identify which dependency must become explicit rather than repair each worker by hand.
Asked what he liked most about the course, Leo pointed to its examples:
It was jam packed full of useful code snippets and examples
— Leo, Senior Software Developer at Matillion.
Tell us how Airflow executes tasks today, which Kubernetes execution approach you are considering, and who will own worker images and operations. We will recommend exercises for that transition.