Spark Driver

Lesson objectives

In this lesson, we will explain the following topics:

• Learn about the role and key functions of the Spark driver.
• Understand how the driver schedules and distributes tasks to executors.
• Explore the communication and resource management responsibilities of the driver.

Spark Driver

The driver is the process in the driver seat.¹ of your Spark Application.

Spark Driver: Key functions

• It transforms all the Spark operations into DAG computations, schedules them, and distributes their execution as tasks across the Spark executors.
• Controlling the execution of a Spark Application.

Spark Driver: Key functions

• Acting as a process on a physical machine, responsible for the overall state of the application on the cluster.
• It instantiates the SparkSession.
• It requests resources (CPU, memory, etc.) from the cluster manager for Spark’s executors (JVMs).
• Once the resources are allocated, it communicates directly with the executors.

Spark Driver: Recap

Watch on Youtube

Watch on our Servers

You can download the videog the link and chose save link as: Download Video