Problem Description


Ingestion job fails with the below zookeeper exception when Infoworks is configured to make hive connections via Zookeeper


[INFO] 2018-11-08 10:06:46,401 [main-EventThread]org.apache.zookeeper.ClientCnxn$EventThread:524 :: EventThread shut down


[ERROR] 2018-11-08 10:06:46,403 [main] infoworks.tools.hive.HiveUtils:618:: Error while trying to execute hive queries java.sql.SQLException: Could not open client transport for any of the Server URI's in ZooKeeper: Unable to read HiveServer2 uri from ZooKeeper


at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:219)


Cause


The issue occurs if the  hive jars in parquet-support are in conflict with the hive jars in the hive client directory (/usr/hdp/current/hive-client/lib)


Solution


Perform the below steps to resolve this issue.


1) Remove /home/dev/infoworks/lib/parquet-support/* from the iw_jobs_classpath
2. Append /usr/hdp/current/hive-client/lib/* to iw_jobs_classpath
3. Add /home/dev/infoworks/lib/parquet-support/parquet-hadoop-bundle-1.6.0.jar to the end of iw_jobs_classpath and then start the ingestion.


The sample iw_jobs_classpath should look like this.


iw_jobs_classpath=/home/dev/infoworks/lib/schemacrawler-12.06.03-main/schemacrawler-12.06.03.jar:/home/dev/infoworks/lib/mongodb/mongo-java-driver-2.12.3.jar:/home/dev/infoworks/bin/tools.jar:/home/dev/infoworks/lib/exec/commons-exec-1.2.jar:/home/dev/infoworks/lib/teradata/tdgssconfig.jar:/home/dev/infoworks/lib/teradata/terajdbc4.jar:/home/dev/infoworks/lib/mongodblogger/logger.jar:/home/dev/infoworks/lib/mongodblogger/log4mongo-java-0.7.4.jar:/home/dev/infoworks/lib/antlr/:/home/dev/infoworks/lib/jackson/:/home/dev/infoworks/lib/jsqlparser/:/home/dev/infoworks/lib/shared/:/home/dev/infoworks/lib/jwt/:/hdpapps/connectors/TDCH/teradata-connector-1.4.4.jar:/etc/hive/conf:/usr/hdp/current/hive-client/lib/:usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-streaming.jar:/home/dev/infoworks/lib/ignite/*:/home/dev/infoworks/lib/parquet-support/parquet-hadoop-bundle-1.6.0.jar