PROBLEM DESCRIPTION


The source table export to BigQuery fails with the below error during the export job in Infoworks’ Autonomous Data Engine (ADE) 2.6.1



ROOT CAUSE


This issue occurs if the Service Account Type is configured to Custom Service Account and if the service account provided in the JSON credentials file does not have access to the GCS Temporary Storage Path (bucket name) provided in the export configurations.



Sample BigQuery export configurations




SOLUTION


Log into Infoworks edge node ad run the below command to check the access to the GCP bucket.


hadoop fs -ls <GCS Temporary Storage Path>. If the command output says Error accessing bucket: <bucket name>


Then login to the GCP cloud console and check if the Requester Pays off setting is turned off for that bucket. The Requester Pays Configuration feature enables you to configure a Google Cloud Storage bucket to indicate that the requester will pay all costs related to accessing the bucket and its objects.



This setting should be turned on for Infoworks to access this bucket from GCP. If this setting is turned off, turn it on and and then run the export to BigQuery job.


IWX version: 2.6.1


Internal Note: Screen shots provided here are from Macy's environment. This document is strictly for internal use. Please check internally with (Aditya) before sharing any of this content with customers.