Problem Description


Ingestion job for a source fails with the below error in the logs


[ERROR] 2018-04-25 08:48:01,589 [main] infoworks.tools.hive.HiveUtils:607 :: Error while trying to execute hive queries java.lang.ClassNotFoundException: org.apache.hive.jdbc.HiveDriver

at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at infoworks.tools.hive.HiveUtils.getConnection(HiveUtils.java:575)

at infoworks.tools.hive.HiveUtils.getConnection(HiveUtils.java:563)

at infoworks.tools.hive.HiveUtils.execStmts(HiveUtils.java:453)

at infoworks.tools.hive.HiveUtils.dropEntityIfExists(HiveUtils.java:1172)

at infoworks.tools.hive.HiveUtils.dropTableIfExists(HiveUtils.java:1181)

at infoworks.discovery.dbcrawler.rdbms.CrawlController.cleanFoldersAndHiveForTable(CrawlController.java:1284)

at infoworks.discovery.dbcrawler.rdbms.CrawlController.importDB(CrawlController.java:700)

at infoworks.discovery.incremental.CDCMain.main(CDCMain.java:44)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)


Root cause


This issue occurs if the hive-jdbc.jar is not added to the iw_jobs_classpath in conf.properties file.


Solution


a) Go to $IW_HOME/conf directory and open the conf.propeties file.

b) Append the absolute path for the hive-jdbc-2.1.1.jar to the iw_jobs_classpath as shown below and then run the ingestion job.


iw_jobs_classpath=/opt/infoworks/lib/schemacrawler-12.06.03-main/schemacrawler-12.06.03.jar:/opt/infoworks/lib/mongodb/mongo-java-driver-2.12.3.jar:/opt/infoworks/bin/tools.jar:/opt/infoworks/lib/exec/commons-exec-1.2.jar:/opt/infoworks/lib/teradata/tdgssconfig.jar:/opt/infoworks/lib/teradata/teradata-connector-1.1.1.jar:/opt/infoworks/lib/teradata/terajdbc4.jar:/opt/infoworks/lib/mongodblogger/logger.jar:/opt/infoworks/lib/mongodblogger/log4mongo-java-0.7.4.jar:/opt/infoworks/lib/antlr/*:/opt/infoworks/lib/jackson/*:/opt/infoworks/lib/jsqlparser/*:/opt/infoworks/lib/shared/*:/opt/infoworks/lib/jwt/*:/usr/local/hive/conf/:/usr/local/hive/lib/*:/usr/local/hive/lib/hive-serde-2.1.1.jar:/usr/local/hive/lib/hive-metastore-2.1.1.jar:/usr/local/hive/lib/hive-common-2.1.1.jar:/usr/local/hive/lib/libthrift-0.9.3.jar:/usr/local/hive/lib/httpclient-4.4.jar:/usr/local/hive/lib/httpcore-4.4.jar:/usr/local/hive/lib/hive-cli-2.1.1.jar:/usr/local/hive/lib/hive-service-2.1.1.jar:/usr/local/hive/lib/hive-service-rpc-2.1.1.jar:/usr/local/hive/lib/hive-jdbc-2.1.1.jar:/usr/local/hive/hcatalog/share/hcatalog/*:/usr/lib/hive/conf:/opt/infoworks/lib/commons-lang3/*


c) hive-jdbc-2.1.1.jar is for Hive 2.1.1. Use the relevant jar for the hive version that you are using.


IWX versions : 2.3.x and 2.4.x


-Aditya