Description: Set the below-advanced configuration to remove duplicate records during full load ingestion. This can be set to either the source or at table level advanced configuration.

Note: This configuration would be useful when duplicate records in the source system/dataset need to be removed before ingesting them into the target hive table (during full load).

key: should_run_de_dup_on_full_load
value: true

Applicable Infoworks versions: