Problem Description:

Ingestion job failing with Error: java.lang.NullPointerException at org.apache.sqoop.mapreduce.IWSequenceFileImportMapper.validateDecimalColumns in mapper phase. Sample stack trace looks like below.

[ERROR] 2021-05-31 11:05:14,014 [pool-3-thread-1] infoworks.tools.hadoop.mapreduce.IWJob:282 :: AttemptID:attempt_1621841064343_2054_m_000000_0 Info:Error: java.lang.NullPointerException
    at org.apache.sqoop.mapreduce.IWSequenceFileImportMapper.validateDecimalColumns(IWSequenceFileImportMapper.java:208)
    at org.apache.sqoop.mapreduce.IWSequenceFileImportMapper.map(IWSequenceFileImportMapper.java:171)
    at org.apache.sqoop.mapreduce.IWSequenceFileImportMapper.map(IWSequenceFileImportMapper.java:56)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:60)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)


Root cause:

This happens when the column type is decimal but all the data of that column is null in the source end.


Solution:

To fix this we need to recrawl the metadata and change the column type to string for that problematic column in the edit schema page of infoworks UI.


Applicable IWX versions:

IWX 3.1.2 onwards