Infoworks

Problem Description:

Kafka streaming ingestion fails with below error

Caused by: java.lang.NoSuchMethodError: com.univocity.parsers.csv.CsvFormat.setDelimiter(Ljava/lang/String;)V

    at infoworks.tools.filecrawler.generic.parser.csv.UnivocityCSVParser.<init>(UnivocityCSVParser.java:54)

    at infoworks.tools.filecrawler.generic.parser.csv.FlatFileParserFactory.getParser(FlatFileParserFactory.java:35)

    at infoworks.tools.filecrawler.generic.parser.csv.FlatFileParserFactory.getParser(FlatFileParserFactory.java:141)

    at consumers.spark.message.handlers.DelimitedMessageHandler.getValueMap(DelimitedMessageHandler.java:85)

    at consumers.spark.message.mappers.DelimitedMessageMapper.parseRecords(DelimitedMessageMapper.java:195)

    at consumers.spark.message.mappers.DelimitedMessageMapper.call(DelimitedMessageMapper.java:179)

Root Cause:

The issue happens when the univocity CSV parser is not able to parse the streaming data

Solution:

Configure streaming job to use COMMONS CSV parser instead of UNIVOCITY 

Step 1: Make sure to have commons-csv-1.1.jar in the spark_streaming_job_classpath  for the file $IW_HOME/conf/conf.properties 

Step 2: Restart platform, restapi, queryservice.

cd $IW_HOME/bin

./stop.sh platform queryservice restapi

./start.sh platform queryservice restapi

Step 3: At the source configuration, add advanced configuration 

CSV_PARSER_LIB = COMMONS

Rerun the streaming ingestion job again after making the above changes. 

Applicable Infoworks DataFoundry Versions:

v.2.7, 2.8, 2.9, 3 x