SPARK_MAJOR_VERSION not set

Most of the times when we are accessing spark-shell from command line we see this error message you can get read of this message by simply settings on shell variable

export SPARK_MAJOR_VERSION=2 

Once this variable is set run your spark-shell command again and it will pick up correct spark version and will run fine.

Leave a Comment