Executor, Job, Stage, Task & RDD Persistence

This page gives a hands-on to understand Executor, Job, Stage & Task using the Spark Web App

Let us start an Application. For this demo, Scala shell acts as a Driver (Application)

mountain@mountain:~/sbook$ spark-shell

Connect to web app(localhost:4040) and explore all the tabs. Except for Environment & Executors tab all other tabs are empty

That clearly indicates we have an Executor running in the background to support our Application.

Let us do some action and see what happens,

The First Run

scala> val data = sc.parallelize(1 to 10)
data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:21

scala> data.count
res0: Long = 10     

Let us check all the tabs,

Jobs Tab

We are able to see how our action(count) run is bifurcated into sub components (Job -> Stages -> Tasks). So any action is converted into Job which in turn is again divided into Stages, with each stage having its own set of Tasks.

Given below is the snapshot of other Tabs which are self explanatory

Stages Tab

Executors Tab

The Second run

Let us once again run the action 'count' and see what happens. This time we got a new Job(Job Id 1) with its own Stages & Tasks

scala> data.count
res1: Long = 10

Jobs Tab

Note : Clicking on Description of each Job will take us to Stages specific to that Job

Stages Tab

Note : Clicking on Description of each Stage will provide us information on Tasks related to that Stage

Executors Tab

Adding Persistence

Let us explore the concept of Persistence using Web App. Let us Persist to our RDD 'data' and run action 'count' again. This time we are interested in Storage Tab(It has been empty so far)

scala> import org.apache.spark.storage.StorageLevel

import org.apache.spark.storage.StorageLevel

scala> data.persist(StorageLevel.MEMORY_ONLY)

res3: data.type = ParallelCollectionRDD[0] at parallelize at <console>:21

scala> data.count

res4: Long = 10

The tab provides us information on RDDs that are persisted. The link in RDD Name provides us more information on the RDD

The Executor tab also provides us information on the Memory that is used to persist data(Earlier  memory usage is 0.0 B)

That completes our discussion

Line Count using Scala Shell

In this example, we will count the number of lines in a Text File. Create a text file with random Content
mountain@mountain:~$ cat data.txt
line 1
line 2

Open Scala Shell and execute the following Statements

mountain@mountain:~$ spark-shell

...
Spark context available as sc (1)
scala> val lines = sc.textFile("data.txt") (2)

...

lines: org.apache.spark.rdd.RDD[String] = data.txt MapPartitionsRDD[1] at textFile at <console>:21

scala> lines.count() (3)

...

res0: Long = 2

scala> lines.first()

...

res1: String = line 1

You can exit from shell using,
scala> exit

Controlling the Logs

You must have realized by now, the Scala Shell prints overwhelming amount of logs which you might not need all the time. The amount of logs printed on the Shell can be controlled by setting the property log4j.rootCategory in file log4j.properties

In $SPARK_HOME/conf,
mountain@mountain:~/sk/conf$ cp log4j.properties.template log4j.properties
mountain@mountain:~/sk/conf$ vi log4j.properties
log4j.rootCategory=WARN, console

Setting the log level to WARN will reduce the amount of logs printed on the Shell. Let us dissect the output from our simple program and grasp few important concepts of Spark

Driver

A Driver Program is akin to main() method in programming languages like C, C++ & java. The Driver Program which is part of a Spark Application launches the Application into Spark Cluster. In our case, The Scala Shell acts as a Driver Program

Spark Context (sc)

In Spark, we access the cluster through object of type SparkContext. The Scala shell by default provides the variable sc(1) which is of type SparkContext 

Resilient Distributed Datasets(RDD)

RDD is the basic unit of Data in Spark upon which all Operations are done. In the example above, lines(2) is an RDD. count()(3) is one of the Operation we have performed on the RDD

An RDD Operation can be any one of this type, actions or transformations.

action returns result to the Driver Program or write it to the Storage. An action normally starts a Computation to provide result and always return some other data type other than RDD

transformation returns Pointer to new RDD

Check the link below for some of the common actions & transformations

https://spark.apache.org/docs/latest/programming-guide.html#actions
https://spark.apache.org/docs/latest/programming-guide.html#transformations

In a typical cluster environment an operation is normally parallelized in multiple Nodes.