Description: Set the below-advanced configuration to remove duplicate records during full load ingestion. This can be set to either the source or at table level advanced configuration.


Note: This configuration would be useful when duplicate records in the source system/dataset need to be removed before ingesting them into the target hive table (during full load).


key: should_run_de_dup_on_full_load
value: true


Applicable Infoworks versions:

IWX-5.0