Problem Description


For a JSON ingested table we have an option to delete the hive table data. If you go the hive table>Actions>Delete Records by providing the select query, the delete job would launch a MapReduce job to run the delete query on hive.

Sometimes this MapReduce job will not be launched to delete the records from hive.


Cause


This issue happens if the Advanced configuration INCREMENTAL_APPEND_MODE is set to true at the source level. This is a limitation in Infoworks ADE and there is a bug request(IPD-7061) raised to address this issue in next releases.


 

Solution


Remove the INCREMENTAL_APPEND_MODE from the source advanced config and run the Delete Records job, it is launching an MR job to run the delete records.


Note: INCREMENTAL_APPEND_MODE is actually used for RDBMS ingestion but not from DFI sources.


Applies to Infoworks ADE Version:


v2.3.x, v2.4.x, v2.5.x, v2.6.x, v2.7.x