Using Jupyter Lab Interface¶

As part of our labs, you can learn Spark using Jupyter based interface.

Make sure you are using right kernel Pyspark 2 (top right corner of the notebook).
Use below code to start the Spark Session object so that you can learn Spark in interactive fashion.

from pyspark.sql import SparkSession

import getpass
username = getpass.getuser()

spark = SparkSession. \
    builder. \
    config('spark.ui.port', '0'). \
    config("spark.sql.warehouse.dir", f"/user/{username}/warehouse"). \
    enableHiveSupport(). \
    appName(f'{username} | Getting Started'). \
    master('yarn'). \
    getOrCreate()

Mastering Pyspark

Using Jupyter Lab Interface¶