Problem Description:


Pipeline build job on Infoworks Azure setup fails with the below error.


main]:[06:33:19,966] [DEBUG] [AwbUtil] (AwbUtil.java:151) - stacktrace org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Failed to read external resource hdfs:///user/infoworks-user/temp//udfs//opt/infoworks/df/udfs/df-shared.jar

    at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:255)

    at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:241)

    at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:254)

    at org.apache.hive.jdbc.HivePreparedStatement.execute(HivePreparedStatement.java:98)

    at io.infoworks.awb.utils.shared.SharedConnection.prepareConnection(SharedConnection.java:238)

    at io.infoworks.awb.utils.shared.SharedConnection.getConnection(SharedConnection.java:209)

    at io.infoworks.awb.utils.shared.SharedConnection.executeSharedConnection(SharedConnection.java:170)

    at io.infoworks.awb.utils.shared.SharedConnection.execute(SharedConnection.java:112)

    at io.infoworks.awb.utils.shared.SharedConnection.execute(SharedConnection.java:156)


Root cause: 


This issue happens as the df-shared.jar will be stored in Azure Storage Blob but Infoworks application will try to access it from hdfs file system.


Solution: 


Perform the below steps to add the wasb location in conf.properties file and then trigger the pipeline build job.


a) Login to Infoworks Edge node.

b) Go to /opt/infoworks/conf directory.

c) Open conf.properties file and add the below config at the end of the file.

iw_hdfs_prefix=wasb://techm-infoworks-2019-07-10t12-00-36-396z@infowork16c0f30e3d3.blob.core.windows.net/


Look for the below message in the pipeline build log to get the wasb URI.


-----Message from pipeline build log-----


[main]:[06:33:19,812] [TRACE] [IWFileSystem] (IWFileSystem.java:33) - FileSystem created org.apache.hadoop.fs.azure.NativeAzureFileSystem@7b4a0aef with uri wasb://techm-infoworks-2019-07-10t12-00-36-396z@infowork16c0f30e3d3.blob.core.windows.net


----------------------------------------------


Applicable Versions


Infoworks Azure 2.7.1