At times end users might need to tune certain Spark configurations to support certain workloads or launch clusters with additional spark configurations. 


On Infoworks version 5.2 and above, the custom Spark configuration can be configured through the Cluster template.


1. Navigate to Admin > Manage Enviornaments > Choose the Environment > Click on the Compute tab

2. Expand the 'Advanced Configuration' section  and click on 'Add'


Key: iw_environment_cluster_spark_config

Value: (semicolon separated configuration key-value pairs)


Example


Key: iw_environment_cluster_spark_config

Value: spark.driver.cores=3;spark.executor.cores=3;spark.executor.memory=1g






Spark configurations can also be passed as arguments for individual jobs such as ingestion, sync to target, and  pipeline 


Source/Table advanced configuration:

key:    iw_spark_app_conf
value: (semicolon separated configuration key-value pairs)

Pipelines advanced configuration   :

key:dt_batch_sparkapp_settings
value: (semicolon separated configuration key-value pairs)

Applicable Versions: Infoworks v5.3 and above