How to add labels for Dataproc Ephemeral clusters launched by Ingestion jobs?
Modified on: Fri, 13 May, 2022 at 9:21 AM
I would like to add labels to Dataproc Cluster launched by Infowokrs ingestion jobs
Infoworks provides a pre ingestion job hook that can be used to run a bash script before beginning the ingestion job.
In the below steps, we would leverage the pre ingestion job hook to add labels for Dataproc clusters once they are launched
1. Create a bash script as below
if ! grep -q interactive "/proc/sys/kernel/hostname"
gcloud dataproc clusters update $cluster_name --update-labels env=prod,source=csv --region=us-central1
echo "Added labels to DataProc Cluster"
echo "Interactive Cluster, not adding labels"
Declare your own vairables
Replace with your actual region for the Dataproc Cluster
1. The above script updates labels only for ephemeral clusters launched by ingestion jobs
2. A pre ingestion job hook is applied for all tables in the source and cannot be applied individually for table
Infoworks 5.0, 5.1.X
Did you find it helpful?
Sorry we couldn't be helpful. Help us improve this article with your feedback.