Problem Description


Metadata Crawl for a Parquet Source fails with the below error in the job log.


Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: io/infoworks/saas/ingestion/commons/exception/IWException
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)


Root Cause 


This issue happens if /opt/infoworks/lib/ingestion/commons.jar is missing in the ingestion_databricks_connect_classpath in conf.properties file.


Solution


a) Append /opt/infoworks/lib/ingestion/commons.jar to the beginning of ingestion_databricks_connect_classpath  in $IW_HOME/conf/conf.properties file. This is under the assumption that /opt/infoworks is the home directory where Infoworks Data Foundry is installed.

b) cd $IW_HOME/bin
c) ./stop.sh ingestion-service
d) source env.sh
e) ./start.sh ingestion-service
f) Run the Parquet ingestion. This should resolve the issue.


Applicable Infoworks Data Foundry on Databricks Versions:


v3.2.x