During the pipeline build process we will run the row count query on the pipeline target table to show the record count in the job execution progress.
[awb-t-0]:[19:32:20,359] [DEBUG] [HiveJdbcSession] (HiveJdbcSession.java:40) - Executing statement: SELECT COUNT(*) AS `ROW_COUNT` FROM `osipi_hv`.`historianosimapr_archive`
Sometimes, this query might tale more time if the execution engine for the pipeline is mr.
MR as an execution engine is slower when there are multiple files and folders. Tez implementation has optimized the aggregation. So most likely the aggregation queries should not take much time on tez.
We can perform the below steps to improve the job performance.
a) Disable this row count query execution
Set the execution engine as Tez by setting the below advanced configuration.
Applicable Infoworks Versions: