Problem Description


Ingestion job for a source fails with the below error in the logs


[ERROR] 2018-04-26 08:53:24,465 [main] infoworks.discovery.dbcrawler.rdbms.CrawlController:753 :: Error while creating crawl Threads.

java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

at org.apache.hive.jdbc.HiveConnection.createUnderlyingTransport(HiveConnection.java:418)

at org.apache.hive.jdbc.HiveConnection.createBinaryTransport(HiveConnection.java:438)

at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:225)

at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:182)

at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)

at java.sql.DriverManager.getConnection(DriverManager.java:664)

at java.sql.DriverManager.getConnection(DriverManager.java:247)

at infoworks.tools.hive.HiveUtils$1.run(HiveUtils.java:602)

at infoworks.tools.hive.HiveUtils$1.run(HiveUtils.java:599)


Root cause

This issue occurs if the hive-common-2.1.1.jar is not added to the iw_jobs_classpath in conf.properties file.


Solution

a) Go to $IW_HOME/conf directory and open the conf.propeties file.

b) Append the absolute path for hive-common-2.1.1.jar to the iw_jobs_classpath as shown below and then run the ingestion job again.


iw_jobs_classpath=/opt/infoworks/lib/schemacrawler-12.06.03-main/schemacrawler-12.06.03.jar:/opt/infoworks/lib/mongodb/mongo-java-driver-2.12.3.jar:/opt/infoworks/bin/tools.jar:/opt/infoworks/lib/exec/commons-exec-1.2.jar:/opt/infoworks/lib/teradata/tdgssconfig.jar:/opt/infoworks/lib/teradata/teradata-connector-1.1.1.jar:/opt/infoworks/lib/teradata/terajdbc4.jar:/opt/infoworks/lib/mongodblogger/logger.jar:/opt/infoworks/lib/mongodblogger/log4mongo-java-0.7.4.jar:/opt/infoworks/lib/antlr/*:/opt/infoworks/lib/jackson/*:/opt/infoworks/lib/jsqlparser/*:/opt/infoworks/lib/shared/*:/opt/infoworks/lib/jwt/*:/usr/local/hive/conf/:/usr/local/hive/lib/*:/usr/local/hive/lib/hive-serde-2.1.1.jar:/usr/local/hive/lib/hive-metastore-2.1.1.jar:/usr/local/hive/lib/hive-common-2.1.1.jar:/usr/local/hive/lib/libthrift-0.9.3.jar:/usr/local/hive/lib/httpclient-4.4.jar:/usr/local/hive/lib/httpcore-4.4.jar:/usr/local/hive/lib/hive-cli-2.1.1.jar:/usr/local/hive/lib/hive-service-2.1.1.jar:/usr/local/hive/lib/hive-service-rpc-2.1.1.jar:/usr/local/hive/lib/hive-jdbc-2.1.1.jar:/usr/local/hive/hcatalog/share/hcatalog/*:/usr/lib/hive/conf:/opt/infoworks/lib/commons-lang3/*


c) hive-common-2.1.1.jar is for Hive 2.1.1. Use the relevant jar for the hive version that you are using.


IWX versions : 2.3.x and 2.4.x


-Aditya