Demo: Running Spark on MacOS

Lesson objectives

In this lesson, we will explain the following topics:

• Demonstrate the process of installing and running Spark on macOS.
• Understand the configuration steps required for Spark installation on macOS.
• Explore the execution of Spark applications on a macOS environment.

Apache Spark Installation on macOS

1. Install Homebrew (if not already installed)

Homebrew is a package manager for macOS. If you don’t have it installed, you can install it using:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Follow the on-screen instructions to complete the installation.

2. Install Java

Apache Spark requires Java. Install OpenJDK using Homebrew:

brew install openjdk@11

Add OpenJDK to your PATH:

echo 'export PATH="/usr/local/opt/openjdk@11/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

Verify the installation:

java -version

3. Download Apache Spark 3.4.3

Go to the Apache Spark download page and download the Spark 3.4.3 pre-built for Hadoop 3.3 and later. Alternatively, you can use the following wget command:

wget https://archive.apache.org/dist/spark/spark-3.4.3/spark-3.4.3-bin-hadoop3.tgz

4. Extract the Spark tar file

Extract the downloaded tar file:

tar xvf spark-3.4.3-bin-hadoop3.tgz

5. Move Spark to the installation directory

Move the extracted Spark folder to /usr/local/spark:

sudo mv spark-3.4.3-bin-hadoop3 /usr/local/spark

6. Set up environment variables

Open your .zshrc file (or .bashrc if you are using bash):

nano ~/.zshrc

Add the following lines at the end:

export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Save and close the file. Then, apply the changes:

source ~/.zshrc

7. Start Spark

To start the Spark shell, you can run:

spark-shell

Watch on Youtube

Watch on our Servers

You can download the videog the link and chose save link as: Download Video