Problem Description


Metadata crawl for Teradata source fails with the below error.


[ERROR] 2018-04-25 08:35:52,339 [main] infoworks.discovery.dbcrawler.rdbms.CrawlController:161 :: CRAWLING Failed due to org/apache/thrift/TException
java.lang.NoClassDefFoundError: org/apache/thrift/TException
at infoworks.discovery.dbcrawler.rdbms.CrawlController.importSchema(CrawlController.java:394)
at infoworks.discovery.dbcrawler.rdbms.utils.DBCrawler.getTablesAndViews(DBCrawler.java:189)
at infoworks.discovery.dbcrawler.rdbms.CrawlController.crawlSchema(CrawlController.java:252)
at infoworks.discovery.dbcrawler.rdbms.CrawlController.doCrawl(CrawlController.java:121)
at infoworks.discovery.main.Main.startCrawl(Main.java:64)
at infoworks.discovery.main.Main.main(Main.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 12 more


Root cause


This issue occurs if the hive related libthrift-0.9.3.jar is not added to the iw_jobs_classpath in conf.properties file.


Solution


a) Go to $IW_HOME/conf directory and open the conf.properties file.

b) Append the absolute path for the hive libthrift-0.9.3.jar to the iw_jobs_classpath as shown below and then run the metadata crawl.


iw_jobs_classpath=/opt/infoworks/lib/schemacrawler-12.06.03-main/schemacrawler-12.06.03.jar:/opt/infoworks/lib/mongodb/mongo-java-driver-2.12.3.jar:/opt/infoworks/bin/tools.jar:/opt/infoworks/lib/exec/commons-exec-1.2.jar:/opt/infoworks/lib/teradata/tdgssconfig.jar:/opt/infoworks/lib/teradata/teradata-connector-1.1.1.jar:/opt/infoworks/lib/teradata/terajdbc4.jar:/opt/infoworks/lib/mongodblogger/logger.jar:/opt/infoworks/lib/mongodblogger/log4mongo-java-0.7.4.jar:/opt/infoworks/lib/antlr/*:/opt/infoworks/lib/jackson/*:/opt/infoworks/lib/jsqlparser/*:/opt/infoworks/lib/shared/*:/opt/infoworks/lib/jwt/*:/usr/local/hive/conf/:/usr/local/hive/lib/*:/usr/local/hive/lib/hive-serde-2.1.1.jar:/usr/local/hive/lib/hive-metastore-2.1.1.jar:/usr/local/hive/lib/hive-common-2.1.1.jar:/usr/local/hive/lib/libthrift-0.9.3.jar:



c) libthrift-0.9.3.jar is for Hive 2.1.1. Use the libthrift jar for the hive version that you are using.


IWX versions : 2.3.x and 2.4.x