Problem Description


Spark pipeline job fails with the below error in the UI.


Error:"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState


Root cause :This issue happens if the spark-shell is not able to locate the hive-site.xml from /opt/mapr/spark/spark-2.2.1/conf directory and so it will not be able to connect to Hive server.


Solution: 

a) Go to $IW_HOME/conf directory.

b) Open con.properties file 

c) Add /opt/mapr/spark/spark-2.2.1/conf to the beginning of df_tomcat_classpath and df_batch_classpath


df_batch_classpath=/opt/mapr/spark/spark-2.2.1/conf/:/opt/infoworks/lib/extras/dt/*:/opt/infoworks/df/udfs/*:/opt/infoworks/df/apache-tomcat-8.0.33/lib/*:/opt/infoworks/bin/df-commons.jar:/opt/infoworks/bin/tools.jar:/opt/infoworks/lib/spark-jackson/*:/opt/infoworks/lib/df/*:/opt/infoworks/lib/mongodb/mongo-java-driver-3.8.0.jar:/opt/infoworks/lib/shared/*:/opt/infoworks/platform/bin/notification-common.jar:/opt/infoworks/platform/bin/platform-common.jar:/opt/mapr/spark/spark-2.2.1/jars/*:/opt/mapr/hive/hive-2.1/lib/*:/opt/mapr/hive/hive-2.1/conf/

df_tomcat_classpath=/opt/mapr/spark/spark-2.2.1/conf/:/opt/infoworks/lib/extras/dt/*:/opt/infoworks/df/udfs/*:/opt/infoworks/df/apache-tomcat-8.0.33/lib/*:/opt/infoworks/bin/df-commons.jar:/opt/infoworks/bin/tools.jar:/opt/infoworks/lib/spark-jackson/*:/opt/infoworks/lib/df/*:/opt/infoworks/lib/mongodb/mongo-java-driver-3.8.0.jar:/opt/infoworks/lib/shared/*:/opt/infoworks/platform/bin/notification-common.jar:/opt/infoworks/platform/bin/platform-common.jar:/opt/mapr/spark/spark-2.2.1/jars/*:/opt/mapr/hive/hive-2.1/lib/*:/opt/mapr/hive/hive-2.1/conf/

 

d) stop.sh df

e) start.sh df


This should resolve the spark pipeline execution issue.