Problem Description


DFI ingestion job fails with the below error when Infoworks edge node tries to get the status of the mapreduce job from Job History server once the mapreduce job is successful.


java.io.IOException: java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for infoworks-user@NP.DDL.MCYBLM.NET to XXXXXXXXXX/XXXXXXXX:10020; Host Details : local host is: "XXXXXXXX/XXXXXXX"; destination host is: "XXXXXXXXX":10020; 

    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:344)

    at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:381)

    at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:626)

    at org.apache.hadoop.mapreduce.Job$7.run(Job.java:761)

    at org.apache.hadoop.mapreduce.Job$7.run(Job.java:758)

    at java.security.AccessController.doPrivileged(Native Method)

    at javax.security.auth.Subject.doAs(Subject.java:422)

    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)

    at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:758)

    at infoworks.discovery.filecrawler.generic.mapreduce.TextCrawlMR.getErrorRecords(TextCrawlMR.java:244)


Cause


This is an exception coming from Hadoop and not from Infoworks. This issue occurs if the Job History server and the timeline server are not restarted after enabling kerberos from Hadoop end.


Solution


Restart Job history server and time line server from Hadoop end and then run the ingestion job. This should resolve the issue.