Using Jupyter Lab InterfaceΒΆ

As part of our labs, you can learn Spark using Jupyter based interface.

  • Make sure you are using right kernel Pyspark 2 (top right corner of the notebook).

  • Use below code to start the Spark Session object so that you can learn Spark in interactive fashion.

from pyspark.sql import SparkSession
import getpass
username = getpass.getuser()
spark = SparkSession. \
    builder. \
    config('spark.ui.port', '0'). \
    config("spark.sql.warehouse.dir", f"/user/{username}/warehouse"). \
    enableHiveSupport(). \
    appName(f'{username} | Getting Started'). \
    master('yarn'). \