Spark Parallel Processing Tutorial

Welcome to the thirteenth lesson Spark Parallel Processing of Big Data Hadoop Tutorial which is a part of ‘Big Data Hadoop and Spark Developer Certification course’ offered by Simplilearn. This lesson will focus on Spark Paralleling Processing.


After completing this lesson, you will be able to:

  • Explain how RDDs are distributed across a spark cluster

  • Analyze how Spark partitions file-based RDDs

  • Explain how Spark executes RDD operations in parallel

  • Explain how to control parallelization through partitioning

  • Analyze how to view and monitor tasks and stages.

Let’s take a look at how Spark works on cluster mode.

Spark Cluster

A Spark Application on Cluster is explained below.

- Image -Spark Application on Cluster

A user can submit a Spark job using Spark-submit. Once the Spark job is submitted, Sparkcontext driver program will be opened which will then go to Cluster Master Node.

From Cluster Master Node, Containers will be opened in Worker Nodes. Followed by Containers, Executors will open up in Worker Nodes, and these Executors will start interacting with Sparkcontext Driver.

RDDs on Spark Cluster

In RDD, Resilient Distributed Datasets, data is partitioned across Worker Nodes by Spark. You can also control the number of partitions created.

Image- Resilient Distributed Datasets on cluster

-image -partitioning-done-by-spark-on-cluster

Let us understand the partitioning from a single file.

File partitioning: Single Files

Partitions are based on the size of the file. You can also specify the minimum number of partitions required as textFile(file,minPartitions). By default, it will be two partitions when running on a spark cluster. More the number of partitions, more the parallelization.

Now that we have discussed the file partitioning from a single file let us learn about the file partitioning from multiple files.

File partitioning: Multiple Files

Using command sc.textFile(“mydir/*”), each file becomes at least one partition. File-based operations can be done per partition, for example parsing XML.

The next command is sc.wholeTextFiles (“mydir”). This command is used for partitioning multiple small files and also creating a key-value pair RRD, where Key stands for file name and value stands for file contents.

Most RDD operations work on each element of an RDD and the other few works on each partition. Some of the commands that are used for partition are:

  • foreachPartition - It is used for calling a function for each partition.

  • mapPartitions - It is used to create a new RDD by executing a function on each partition in the current RDD.

  • mapPartitionsWithIndex - This is the same as mapPartitions, but this includes an index of the partitions.

Note: Functions for partition operations take iterators. We will look at an example for one of the RDDs for better understanding.

Let us understand foreachPartition with an example.


In the example below, we have created a function printFirstLine which will calculate the first line for each partition.


Let’s assume we already have an RDD created, which is named myrdd. We can pass printFirstLine created function into foreachPartition to calculate the first line for each partition. Now that you have understood the commands used for partitioning, let's understand HDFS and data locality with an example in the next section.

HDFS and Data Locality

In the diagram, you can notice multiple data nodes.

- Diagram -

Now, using hdfs dfs -put mydata, you can push the mydata file to HDFS. Let’s assume that it is saved in HDFS disk in three blocks.

When the data is saved in HDFS disk, you can start programming in Spark. Once you start programming, Spark context will be available, which will open up executor in datanodes.

Using sc.textfile, you can push mydata in Spark executor. Since this is a transformation step, RDD will remain blank.

Using action trigger execution, tasks on executors load data from blocks into partitions. Then the data is distributed across executors until an action returns a value to the driver.

Parallel Operations on Partitions

RDD operations are executed in parallel on each partition. Tasks are executed on the Worker Nodes where the data is stored.

Some operations preserve partitioning, such as map, flatMap, filter, distinct, and so on. Some operations repartition, such as reduceByKey, sortByKey, join, groupByKey, and so on.

Let us learn about the operations in stages.

Operations in stages

Operations that can run on the same partition are executed in stages. Tasks within a stage are pipelined together. Developers should be aware of operational stages to improve performance.

Listed are some of the Spark terminologies:

  • Job: a set of tasks executed as a result of an action. Stage: a set of tasks in a job that can be executed in parallel.

  • Task: an individual unit of work sent to one executor. Application: can contain any number of jobs managed by a single driver.

Now let’s look at how Spark calculates stages.

How Spark calculates stages?

Spark constructs a Directed Acyclic Graph or DAG of RDD dependencies. These dependencies are of two types:

  1. Narrow dependencies, where Each partition in the child RDD depends on just one partition of the parent RDD. No shuffle is required between executors. Nodes, where the RDDs are created, can be collapsed into a single stage Example: map, filter,

  2. Union Wide or shuffle dependencies, where Multiple child partitions depend on each partition in the parent RDD. Wide dependencies define a new stage. Example: reduceByKey, join, groupByKey Let’s go through the process of controlling the level of Parallelism.

“Wide” operations such as reduceByKey partition result in RDDs. The more the number of partitions, the more are the parallel tasks. Spark cluster will be under-utilized if there are too few partitions.

You can control the number of partitions by optional numPartitionsparameter in the function call. We can see Spark application UI from localhost: 4040. We can notice all the Spark jobs in this UI.


Now let's summarize what we've learned in this lesson.

  • RDDs are stored in the memory of Spark executor

  • Java Virtual Machines, JVMs Data is split into partitions.

  • Each partition in a separate executor RDD operations are executed on partitions in parallel

  • Operations that depend on the same partition are pipelined together in stages.

  • Operations that depend on multiple partitions are executed in separate stages.


This concludes the lesson on Spark Parallel Processing. In the next lesson of this Big Data Hadoop tutorial, we will look into “Spark RDD Optimization Techniques.

  • Disclaimer
  • PMP, PMI, PMBOK, CAPM, PgMP, PfMP, ACP, PBA, RMP, SP, and OPM3 are registered marks of the Project Management Institute, Inc.

Request more information

For individuals
For business
Phone Number*
Your Message (Optional)
We are looking into your query.
Our consultants will get in touch with you soon.

A Simplilearn representative will get back to you in one business day.

First Name*
Last Name*
Phone Number*
Job Title*