Problem Description:


Sometimes we might need to set hive configuration variables during the Pipeline export job to BigQuery for below reasons.

a) Tune the export job 

b) Provide the container sizes when the execution engine is set to Tez

c) Provide the Tez queue name etc



Solution:


Set the below Global Advanced configuration under Admin>Configurations section in Infoworks ADE to pass the hive configurations variables during the job execution.


For example,


Key: bq_hive_conf_vars

value: hive.exec.dynamic.partition.mode=nonstrict,hive.execution.engine=tez,hive.tez.container.size=<val>,tez.queue.name=<queuename>;


The values should be separated by ',' (comma) if there are multiple configurations.






Applied to Versions:


IWX ADE v2.4.x,v2.5.x,c2.6.x,v2.7.0