Problem Description:


If you execute a custom sqoop command inside a bash node in a workflow, it fails with the below exception in Infoworks EDO2 v2.8.x


2019-09-11 09:29:32,871] {bash_operator.py:110} INFO - Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatOutputFormat


Root cause:


This issue exists if the HCAT jars are not getting picked up when we run the command from the bash node. 


So when we run the sqoop command from the bash node, IWX will by default launch the shell by sourcing the env.sh file and env.sh file has HCAT_HOME which conflicts with the HCAT jars present in your existing PATH in the edge node.


Solution:


a) The below unset command in the bash node inside the workflow to unset the HCAT_HOME variable so that it will use the default HCAT jars from the PATH variable.

unset HCAT_HOME;/usr/bin/sqoop export --connect 'jdbc:sqlserver://dimasqldb.database.windows.net;database=dimasqldb;' --username dimasqldbadmin --password Pepsico1234! --hcatalog-database nbtest --hcatalog-table nbtest2 --table nbtest2 --update-key roll --update-mode allowinsert -- --schema accolade -m 1


Applicable Infoworks Versions:


v2.4.x,v2.5.x,v2.6.x,v2.7.x,v2.8.x