From Spark on YARN to Kubernetes: training for data teams

We offer private, instructor-led training for data engineers who run Spark applications on YARN and need to move suitable execution workloads to Kubernetes.

Your team understands Spark jobs and their data dependencies. The execution environment changes around Spark, including images, driver and executor placement, credentials, and the evidence available during a failure.

LearnKube connects these responsibilities through a representative Spark application. Proposed Spark-specific exercises adapt the Kubernetes course to submission, data access, and execution diagnosis.

Discuss your team's training

Learning in practice

Hands-on learning and the skills engineers take back to work.

Preview: course-wide figures are not yet available.

Hands-on learning
60%
Of instruction time spent on labs and challenges.
Troubleshooting confidence
86%
Of respondents report greater confidence diagnosing Kubernetes problems.
Relevant to your work
92%
Of respondents say the course addressed their engineering responsibilities.
Skills put into practice
95%
Of respondents applied their new skills at work within 90 days.

Keep Spark behavior clear as the execution platform changes

Submit the application by configuring images, execution mode, and Kubernetes access, so the driver and executors can start with the dependencies they need.
Preserve data access and recovery by examining storage, credentials, and retry ownership, so execution changes do not silently alter required inputs or outputs.
Diagnose stalled execution by correlating Spark status with Pod events and resource constraints, so engineers distinguish application behavior from placement or infrastructure problems.
Define platform responsibilities by documenting submission, runtime images, capacity, and storage ownership, so data and Kubernetes teams know which actions they control.

Spark stays, while resource and execution management change

In YARN cluster mode, the Spark driver runs inside a YARN-managed application master process. In Kubernetes cluster mode, Spark creates a driver Pod that creates executor Pods. The application still needs its libraries, data, and credentials. Engineers must understand where Spark makes execution decisions and where Kubernetes supplies resources and placement.

A healthy driver Pod is not evidence that a Spark stage can make progress. I want engineers to trace executor placement and data access before they change application code to solve a cluster-side failure.
— Daniele Polencic, LearnKube founder and Kubernetes instructor

Current task or requirement	New concept or decision	Practice
Submit a Spark application	Driver and executor Pods	Trace cluster-mode submission
Provide runtime libraries	Images and dependency delivery	Diagnose a missing library
Allocate execution resources	Requests, limits, and placement	Inspect Pending executors
Reach required datasets	Credentials and storage access	Diagnose an unreadable input

How we approach this transition

We propose a Kubernetes workshop customized around one Spark application and its data dependencies. Your team compares its current YARN execution mode with the intended Kubernetes configuration.

Spark-specific work is proposed customization of the course, not a default module. The exercises focus on execution and operations. They do not assume that every Hadoop service or dataset moves with the Spark application.

A recommended four-day workshop for Spark execution teams

This proposed agenda connects the Kubernetes baseline to driver and executor behavior. It keeps application performance tuning separate from the resource and operating-model transition.

Day 1

Images, Pods, and Spark execution

We explain container packaging, Pod lifecycle, and workload controllers before examining a Spark submission. You will distinguish service readiness and deployment rollouts from the completion of a Spark application.

Hands-on exercises

Submit a sample Spark application in Kubernetes cluster mode and locate its driver and executors.
Diagnose a startup failure caused by an image or runtime dependency.

Day 2

Repeatable configuration and controller behavior

You will use Helm and compare Kustomize for supporting Kubernetes resources. We explain the control plane and nodes, while separating Spark submission settings from the resources that support execution.

Hands-on exercises

Prepare repeatable service-account and configuration resources for the sample application.
Stop an executor Pod and inspect how Spark and Kubernetes report the resulting behavior.

Day 3

Connectivity, placement, and resource demand

We trace driver, executor, and data-service connections through Kubernetes networking. You will review DNS, Services, ingress for supporting interfaces, network policies, service mesh implications, and placement constraints.

Hands-on exercises

Diagnose an executor that cannot reach the driver or a required data endpoint.
Inspect a Pending executor whose resource request cannot fit available capacity.

Day 4

Durable data, allocation, and access

You will examine secrets, persistent storage, and the driver's Kubernetes permissions. We distinguish Spark executor allocation from HPA behavior and node capacity, then review authentication and RBAC.

Hands-on exercises

Restore access to a sample dataset after a credential or mount failure.
Compare requested executors with available node resources and identify the limiting component.

Your Spark execution mode, data services, and platform responsibilities can shape the agenda. Get in touch to tailor the workshop to your Spark-on-Kubernetes transition.

Distinguish a Spark problem from a placement problem

When an engineer changes Spark settings because a job appears stalled, the instructor can inspect its executor Pods and scheduler events. A resource-constrained example shows why application diagnosis needs evidence from both execution layers.

Asked what he liked most about a separate LearnKube course, Philip pointed to the prepared exercises:

Presenter, and hands on exercises/labs of covering the material that was in the presentation. Everything, including the VMs were setup from beforehand so that it was seamless to carry out the exercises.
— Philip, Lead Data Engineer at EDF Trading.

Start with one Spark application and its execution path

Tell us how Spark runs on YARN today, which data services it needs, and who will own the Kubernetes environment. We will recommend a course focus and proposed Spark exercises for that setup.

Discuss your team's training