Problem Description: The export of data to BigQuery fails with the below error.


[DEBUG] 2019-09-19 03:50:04,408 [pool-3-thread-1] infoworks.tools.hive.HiveUtils:507 :: Running SQL: CREATE EXTERNAL TABLE `iw_uat_edw_mf_ds`.`FIL_NIL_HOPS_b9548c51c3f32d96f1dac526_bq` (`res_nbr` string,`ln_nbr` string,`ent_ts` string,`added_by_app` string,`sto_nbr` string,`sku_nbr` string,`qty_cur` string,`group_code` string,`grace_days` string,`log_dte` string,`log_reason` string,`cancel_reason` string,`from_fil_loc` string,`from_avail_to_fill` string,`from_avail_to_pick` string,`from_pick_in_progress` string,`from_not_avail_qty` string,`new_fil_loc` string,`new_avail_to_fill` string,`new_avail_to_pick` string,`new_pick_in_progress` string,`new_not_avail_qty` string,`ent_login` string) ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe' LOCATION 'gs://mtech-daas-transact-sata-uat_processing/infoworks/iw_bq_load/37bbe2db-d7dd-4eb5-9ce8-d3c3cbf407dd'
[ERROR] 2019-09-19 03:50:06,257 [pool-3-thread-1] infoworks.tools.hive.HiveUtils:512 :: Error while executing hive queryjava.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.IllegalArgumentException: hadoopPath must not be null)
    at org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:354)
    at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:245)
    at infoworks.tools.hive.HiveUtils.execStmts(HiveUtils.java:508)
    at io.infoworks.gcp.hive.HiveClient.createHiveTable(HiveClient.java:153)
    at io.infoworks.gcp.hive.HiveClient.createTable(HiveClient.java:142)
    at io.infoworks.gcp.bigquery.execution.hive.HiveBQFormatConversionTask.dropAndCreateTable(HiveBQFormatConversionTask.java:143)
    at io.infoworks.gcp.bigquery.execution.hive.HiveBQFormatConversionTask.convertToJson(HiveBQFormatConversionTask.java:62)
    at io.infoworks.gcp.bigquery.execution.hive.HiveBQFormatConversionTask.convertToBQFormat(HiveBQFormatConversionTask.java:58)
    at io.infoworks.gcp.bigquery.execution.hive.HiveBQFormatConversionTask.execute(HiveBQFormatConversionTask.java:53)
    at io.infoworks.gcp.bigquery.execution.hive.HiveBQReplicationJob.execute(HiveBQReplicationJob.java:51)
    at io.infoworks.gcp.bigquery.HiveBQReplicationEngine.replicate(HiveBQReplicationEngine.java:119)
    at infoworks.discovery.export.rdbms.destination.BigQueryDestination.export(BigQueryDestination.java:65)
    at infoworks.discovery.export.ExportWorkerThread.call(ExportWorkerThread.java:36)
    at infoworks.discovery.export.ExportWorkerThread.call(ExportWorkerThread.java:7)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)


Root Cause: This issue occurs if Infoworks is not able to write/access the google storage bucket.


Solution: 

a) Login to Infoworks Edge node using the user who starts the Infoworks services.

b) run hdfs dfs -ls gs://<gcs temporary storage path that you have provided in the config> 

f.e.g hdfs dfs -ls gs://mtech-daas-transact-sata-uat_processing/infoworks

c) Check if <Infoworks_user> is able to access the bucket location and has read/write access. Provide access if it is missing.

d) Run the export job again.