Description: Set the below-advanced configuration to remove duplicate records during full load ingestion. This can be set to either the source or at table level advanced configuration.
Note: This configuration would be useful when duplicate records in the source system/dataset need to be removed before ingesting them into the target hive table (during full load).
key: should_run_de_dup_on_full_load value: true
Applicable Infoworks versions: