PROBLEM DESCRIPTION


Teradata ingestion job will fail with the below error in the job logs.


[ERROR] 2018-12-03 11:20:47,874 [pool-5-thread-1] infoworks.tools.hadoop.mapreduce.IWJob:187 :: AttemptID:attempt_1543252886218_0321_m_000000_0 Info:Error: java.lang.RuntimeException: java.io.IOException: SQLException in nextKeyValue 


POTENTIAL ROOT CAUSE


This issue might occur if you are using lower version of Teradata jdbc driver which is not compatible with the Teradata DB version. Open the job log and look for a stack trace shown below.


Caused by: java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC 15.00.00.20] [Error 804] [SQLState 08S01] Socket communication failure for Packet receive Mon Dec 03 11:18:24 MST 2018 socket orig=devedw1.caremarkrx.net local=0.0.0.0/0.0.0.0:56860 remote=devedw1.caremarkrx.net/204.99.40.60:1025 keepalive=unavailable nodelay=unavailable receive=unavailable send=unavailable linger=unavailable traffic=unavailable concurrent=3 contimeout=10000 conwait=1000 connecttime=1 connecttotaltime=6 connectattempts=1 connectfailures=0 reconnectattempts=0 recoverable=false redrive=false failurecache={} cid=3b9d6699 sess=1713934 java.io.IOException: Bad response message header with invalid message length of 5296792 bytes  at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.read(TDNetworkIOIF.java:738)  at com.teradata.jdbc.jdbc_4.io.TDPacketStream.readStream(TDPacketStream.java:773)  at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:137)  at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)  at com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)  at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:385)  at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:327)  at com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecuteQuery(TDPreparedStatement.java:190)  at com.teradata.jdbc.jdbc_4.TDPreparedStatement.executeQuery(TDPreparedStatement.java:2580)  at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:109)  at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)  at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)  at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)  at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)  at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:60)  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)  at java.security.AccessController.doPrivileged(Native Method)  at javax.security.auth.Subject.doAs(Subject.java:422)  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:94)
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeDriverJDBCException(ErrorFactory.java:69)
at com.teradata.jdbc.jdbc_4.util.ErrorFactory.makeIoJDBCException(ErrorFactory.java:207)
at com.teradata.jdbc.jdbc_4.util.ErrorAnalyzer.analyzeIoError(ErrorAnalyzer.java:59)
at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.read(TDNetworkIOIF.java:889)
at com.teradata.jdbc.jdbc_4.io.TDPacketStream.readStream(TDPacketStream.java:773)
at com.teradata.jdbc.jdbc_4.statemachine.StatementReceiveState.action(StatementReceiveState.java:137)
at com.teradata.jdbc.jdbc_4.statemachine.StatementController.runBody(StatementController.java:123)
at com.teradata.jdbc.jdbc_4.statemachine.PreparedStatementController.run(PreparedStatementController.java:46)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:385)
at com.teradata.jdbc.jdbc_4.TDStatement.executeStatement(TDStatement.java:327)
at com.teradata.jdbc.jdbc_4.TDPreparedStatement.doPrepExecuteQuery(TDPreparedStatement.java:190)
at com.teradata.jdbc.jdbc_4.TDPreparedStatement.executeQuery(TDPreparedStatement.java:2580)
at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:109)
at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
... 12 more
Caused by: java.io.IOException: Bad response message header with invalid message length of 5296792 bytes
at com.teradata.jdbc.jdbc_4.io.TDNetworkIOIF.read(TDNetworkIOIF.java:738)
... 22 more


In this particular case, the Teradata jdbc driver used is of version 15.00 and the Teradata DB version is 16.10. Run the below sql query on the Teradata database to get the exact version.


select * from dbc.dbcinfo;


RESOLUTION


Steps to resolve the issue:


a) Download the required/supported version of terajdbc4.jar from Teradata website. https://downloads.teradata.com/download

b) Remove the old terajdbc4.jar from $IW_HOME/lib/teradata

c) Place the latest/supported terajdbc4.jar that is downloaded

d) Clear the below cached teradata jars from the below hdfs locations. The iw_hdfs_temp_home path is set in conf.properties file


hdfs dfs -rmr <iw_hdfs_temp_home>/iwjobs/libjars/home/dev/infoworks/lib/shared/terajdbc4.jar

hdfs dfs -rmr <iw_hdfs_temp_home>/iwjobs/libjars/home/dev/infoworks/lib/shared/tdgssconfig.jar

hdfs dfs -rmr <iw_hdfs_temp_home>/iwjobs/libjars/home/dev/infoworks/lib/teradata/terajdbc4.jar

hdfs dfs -rmr <iw_hdfs_temp_home>/iwjobs/libjars/home/dev/infoworks/lib/teradata/tdgssconfig.jar

e) Run the ingestion job again (Infoworks service restart is not required).


APPLIES TO


Infoworks ADE versions:2.4.x,2.5.x,2.6.x