Problem Description:


Pipelines that are built with Spark or Hive as the execution engine fails with the below error in the .out file in the job log.


ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7ANTLR Tool version 4.5 used for code generation does not match the current runtime version 4.7ANTLR Runtime version 4.5 used for parser compilation does not match the current runtime version 4.7infoworks.tools.ExceptionHandling.IWRunTimeException: [BATCH_PIPELINE_PLAN_ERROR: 2501]: An error occurred while validating the built pipeline target. These details are available in logs.


Root cause:


IWX Data Foundry uses an open-source tool called ANTLR to parse a YAML file /opt/infoworks/conf/df_executor_configs.yaml to set some configurations and add some jars required for Spark and Hive execution engines during the pipeline build process.

https://www.antlr.org/

This issue happens if the antlr4-runtime-4.7.jar is missing in the df_batch_classpath in conf.properties file or an old version of the antlr jar (4.5)  is getting picked up. This is a known issue in IWX Data Foundry 2.9.1 and it is fixed in v3.1.0.


Solution:


Upgrade IWX Data Foundry to v3.1.0 to fix this issue or perform the below steps as a workaround in the current version that you are using.


a) Login to Infoworks Edge node with the user who starts IWX services.

b) Go to $IW_HOME/conf/conf.properties file.

c) Open it and add the below antlr jar in the df_batch_classpath.


/opt/infoworks/lib/extras/dt/*:/opt/infoworks/df/udfs/*:/opt/infoworks/bin/df-commons.jar:/usr/lib/spark/jars/antlr4-runtime-4.7.jar:/opt/infoworks/lib/df/*:/usr/lib/spark/jars/*:/opt/infoworks/lib/mongodblogger/*:/usr/lib/spark/conf/:/usr/lib/hive/conf/:/opt/infoworks/platform/bin/notification-common.jar:/opt/infoworks/platform/bin/platform-common.jar:/opt/infoworks/platform/lib/notification-client/*:/usr/lib/hive/lib/*


d) Run the Hive/Spark pipeline.



Applicable IWX Data Foundry Versions:


v2.5.x,v2.6.x,v2.7.x,v2.8.x,v2.9.x,v3.0