Problem Description


Full ingestion job fails with the below error in the job log.


[ERROR] 2019-06-19 11:32:36,321 [pool-7-thread-1] infoworks.tools.utils.IWUtil:1617 :: Couldnt switch between temp and orig full load location. Data is crawled though.java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]; Host Details : local host is: "ustry1mety0151l.metnet.net/XX.XX.XX.XXX"; destination host is: "latam-dev-d02.metnet.net":8020; 

 at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:785)

 at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1558)

 at org.apache.hadoop.ipc.Client.call(Client.java:1498)

 at org.apache.hadoop.ipc.Client.call(Client.java:1398)


Cause


The communication between the Infoworks edge node to Hadoop Namenode in a Kerberos enabled Hadoop ecosystem fails with the above mentioned error. This issue occurs if hadoop filesystem is initialised before the renewal of kerberos ticket. This is a limitation in Infoworks ADE v2.7.0 and a bug request is raised to address this issue.


Workaround


Renew the kerberos ticket by running the below kinit command.


kinit -kt <infoworks_keytab_file> infoworks-principle


This issue will be addressed in Infoworks ADE v2.7.2.


Applies to Infoworks ADE Version:


v2.6.x,v2.7.0