Problem Description:


Ingestion job fails with the below error in Infoworks DataFoundry 4.2


{"life_cycle_state":"INTERNAL_ERROR","result_state":"FAILED","state_message":"Library installation failed for library jar: \"dbfs:/infoworks/lib/ingestion-dist/commons.jar\"\n. Error messages:\njava.lang.RuntimeException: ManagedLibraryInstallFailed: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: dbfs:/infoworks/lib/ingestion-dist/commons.jar for library:JavaJarId(dbfs:/infoworks/lib/ingestion-dist/commons.jar,,NONE),isSharedLibrary=false"}



Root Cause: This issue happens is some of the ingestion related jars are not copied properly into the Databricks file system(dbfs) during the DataFoundry Installation process.



Solution: Perform the below steps to copy the jars manually into dbfs and then trigger the ingestion job again.


a) Login to Infoworks Edge node.
b) sudo su infoworks
c) source /opt/infoworks/bin/env.sh;


dbfs cp --overwrite <databricks_iw_home>/lib/ingestion-dist/commons.jar dbfs:<dbfs_iw_home>/lib/ingestion-dist/commons.jar;

dbfs cp --overwrite <databricks_iw_home>/lib/ingestion-dist/pojo.jar dbfs:<dbfs_iw_home>/lib/ingestion-dist/pojo.jar;

dbfs cp --overwrite <databricks_iw_home>/lib/ingestion-dist/dao.jar dbfs:<dbfs_iw_home>/lib/ingestion-dist/dao.jar;

dbfs cp --overwrite <databricks_iw_home>/lib/ingestion-dist/core.jar dbfs:<dbfs_iw_home>/lib/ingestion-dist/core.jar;

dbfs cp --overwrite <databricks_iw_home>/lib/ingestion-dist/json-simple-1.1.1.jar dbfs:<dbfs_iw_home>/lib/ingestion-dist/json-simple-1.1.1.jar;

dbfs cp --overwrite <databricks_iw_home>/lib/ingestion-dist/mongo-java-driver-3.9.1.jar dbfs:<dbfs_iw_home>/lib/ingestion-dist/mongo-java-driver-3.9.1.jar;

dbfs cp --overwrite <databricks_iw_home>/lib/platform/scala-utils/sparkMetrics-assembly.jar dbfs:<dbfs_iw_home>/lib/platform/scala-utils/sparkMetrics-assembly.jar

 


Sample command:


dbfs cp --overwrite /opt/infoworks/lib/ingestion-dist/lib/ingestion-dist/commons.jar dbfs:/infoworks/lib/ingestion-dist/commons.jar;


where <databricks_iw_home> is /opt/infoworks/lib/ingestion-dist and <dbfs_iw_home> is /infoworks. Provide these paths according to your installation.



d) Run the ingestion job again.


Applicable Infoworks DataFoundry Versions:

v4.2