Spark Hbase Kerberos Test Job

To run a test job with spark and hbase

./bin/spark-submit --class org.apache.spark.examples.HBaseTest \
--master yarn --deploy-mode client --files /etc/hbase/conf/hbase-site.xml \
--conf "spark.driver.extraClassPath=/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-.jar:/usr/hdp/current/hbase-client/lib/htrace-core--incubating.jar" \
--conf "spark.executor.extraClassPath=/usr/hdp/current/hbase-client/lib/hbase-common.jar:/usr/hdp/current/hbase-client/lib/hbase-client.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/lib/hbase-protocol.jar:/usr/hdp/current/hbase-client/lib/guava-.jar:/usr/hdp/current/hbase-client/lib/htrace-core--incubating.jar" \
./lib/spark-examples*.jar ambarismoketest


To enable debug on spark

log4j.logger.org.apache.spark.repl=INFO


log4j.logger.org.apache.spark.repl.Main=INFO

To verify if we are able to get Delegation Token

18/06/25 18:13:43 DEBUG YarnSparkHadoopUtil: Added HBase security token to credentials

Leave a Comment