PROBLEM DESCRIPTION


Ingestion job fails with the below error during the map-reduce phase.


[INFO] 2018-12-04 16:05:57,010 [pool-5-thread-1] org.apache.hadoop.hdfs.DFSClient:1052 :: Created HDFS_DELEGATION_TOKEN token 3231 for da_bigdata_iwx_p on ha-hdfs:jp04hdpnn
[ERROR] 2018-12-04 16:05:57,084 [pool-5-thread-1] infoworks.tools.hadoop.mapreduce.IWJob:133 :: Exception occured : java.io.IOException: java.net.ConnectException: Connection refused (Connection refused)
                at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1029)
                at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
                at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2407)
                at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
                at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
                at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
                at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:142)
                at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266)
                at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139)
                at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
                at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:422)


ROOT CAUSE


This issue occurs if the KMS (Hadoop Key Management Server) is down and kerberos is not able to talk to this KMS service to get the delegation token to submit the ingestion map-reduce job.


Infoworks request for token generation with the infoworks user (user who starts Infoworks services) to submit the map reduce job then kerberos will talk to KMS service and check whether it has permission to do that . If yes then a delegation token will be generated and Infoworks will submit the job with that token.


RESOLUTION


a) Check if KMS service is up and running. 


b) Identify the host where KMS server is running , by logging into Ambari (if the Hadoop distribution is Hortonworks Data Platform .aka. HDP) and check the below configuration variable of HDFS.


 service.hadoop.kms.key.provider.uri. 

 

For example, if service.hadoop.kms.key.provider.uri is set to kms://http@bivm.ibm.com:16000/kms , kms is running on bivm.ibm.com


Following is an example on how to perform the check on linux


netstat -anp|grep 16000


Or


c) Check this configuration in etc/hadoop/kms-site.xml file.

d) Check if KMS service is running on that host occupying that port.

e) If the kms is not running start the kms server and then run the ingestion job.


Referred Links


https://community.hortonworks.com/questions/10497/after-enable-the-kerberos-the-spark-does-not-work.html


http://www-01.ibm.com/support/docview.wss?uid=swg21980695



APPLIES TO/ AFFECTED VERSION(S)


Infoworks ADE  v2.4.x, v2.5.x and v2.6.x