Description: Starting from IWX v3.1.2, we can provide the Spark configurations like spark driver memory, executor memory, and any other spark configurations to the interactive pipeline job like the preview data job in the Spark pipeline.
Steps to provide the spark configurations for a pipeline interactive job.
1. Login to IWX Edge node as Infoworks user.
2. Copy the contents of <IW_HOME>/conf/dt_spark_defaults.conf to <IW_HOME>/conf/dt_interactive_spark_defaults.conf (create a file with this namee)
3. Edit memory configurations in <IW_HOME>/conf/dt_interactive_spark_defaults.conf as per your requirement.
For example, if you want to set/modify the Spark driver and executor memories for interactive jobs, set the below values in the dt_interactive_spark_defaults.conf file (Make sure there are no duplicate entries in the file).
spark.executor.memory=1024m
spark.driver.memory=1024m
spark.driver.memory=1024m
4. Add the following configuration in <IW_HOME>/conf/conf.properties
dt_spark_configfile=/opt/infoworks/conf/dt_interactive_spark_defaults.conf
Note: In this case, the IWX application is installed under /opt/infoworks directory. i.e IW_HOME=/opt/infoworks/
Ensure that the owner of the dt_interactive_spark_defaults.conf file is the infoworks user and the file name including the path is provided properly in the conf.properties file.
5. Restart the dt service when there are no jobs running so that the above configuration changes will be picked up.
<IW_HOME>/bin/stop.sh dt
source <IW_HOME>/bin/env.sh
<IW_HOME>/bin/start.sh dt
Applicable IWX Versions:
IWX v3.1.2,v3.3.3