Spark Cluster Manager TypesΒΆ

Let us get an overview of different Spark Cluster Managers on which typically Spark Applications are deployed.

  • Here are the supported cluster manager types.

    • Local (used for development and unit testing).

    • Stand Alone

    • YARN

    • Mesos

  • Here are the popular distributions which use YARN to deploy Spark Applications.

    • Cloudera

    • AWS EMR

    • Google Dataproc

    • Hortonworks

    • MapR

  • Databricks uses Stand Alone for running or deploying Spark Jobs.

Different Spark Modules Launching Spark CLI