Advent of 2021, Day 6 – Setting up IDE

Series of Apache Spark posts:

• Dec 01: What is Apache Spark
• Dec 02: Installing Apache Spark
• Dec 03: Getting around CLI and WEB UI in Apache Spark
• Dec 04: Spark Architecture – Local and cluster mode
• Dec 05: Setting up Spark Cluster

Let’s look into the IDE that can be used to run Spark.

Remember that Spark can be used with languages: Scala, Java, R, Python and each give you different IDE and different installations.

Jupyter Notebooks

Start Jupyter Notebooks and create a new notebook and you can connect to Local Spark installation.

For the testing purposes you can add code like:

spark = SparkSession.builder.set_master("spark://tomazs-MacBook-Air.local:7077")

And start working with the Spark code.

Python

In Python, you can open a PyCharm or Spyder and start working with python code:

import findspark
findspark.init("/opt/spark")
from pyspark import SparkContext

sc = SparkContext(appName="SampleLambda")
x = sc.parallelize([1, 2, 3, 4])
res = x.filter(lambda x: (x % 2 == 0))
print(res.collect())
sc.stop()

R

Open RStudio and install sparkly package, create a context and run a simple R script:

# install
devtools::install_github("rstudio/sparklyr")
spark_disconnect(sc)

# install local version
spark_install(version = "2.2.0")

# Create a local Spark master 
sc <- spark_connec(master = "local")

iris_tbl <- copy_to(sc, iris)
iris_tbl

spark_disconnect(sc)

There you go. This part was fairly short but crucial for coding.

Tomorrow we will start exploring spark code.

Compete set of code, documents, notebooks, and all of the materials will be available at the Github repository: https://github.com/tomaztk/Spark-for-data-engineers

Happy Spark Advent of 2021!