Description: Starting from IWX v3.1.2, we can provide the Spark configurations like spark driver memory, executor memory, and any other spark configurations to the interactive pipeline job like the preview data job in the Spark pipeline.
Steps to provide the spark configurations for a pipeline interactive job.
1. Login to IWX Edge node as Infoworks user.
3. Edit memory configurations in <IW_HOME>/conf/dt_interactive_spark_defaults.conf as per your requirement.
4. Add the following configuration in <IW_HOME>/conf/conf.properties
Note: In this case, the IWX application is installed under /opt/infoworks directory. i.e IW_HOME=/opt/infoworks/
5. Restart the dt service when there are no jobs running so that the above configuration changes will be picked up.
Applicable IWX Versions: