How-to

How to configure custom Spark settings at Cluster template
At times end users might need to tune certain Spark configurations to support certain workloads or launch clusters with additional spark configurations.  ...
Tue, 15 Nov, 2022 at 4:16 AM
How to add a new region for AWS Databricks in Infoworks
To add a new region version for AWS Databricks, perform the following steps: Step 1: SSH into Infoworks machine. Step 2: Navigate to the path where I...
Fri, 24 Jun, 2022 at 1:22 PM
SAML integration with Okta
Fri, 24 Jun, 2022 at 5:25 AM
How to get the tpt.out file for a failed TPT ingestion job in IWX 5.0
Description: During the Teradata TPT ingestion, Infoworks DataFoundry would use the Teradata Parallel Transport Utility installed on the Master node of the ...
Wed, 15 Sep, 2021 at 3:27 PM
How to debug and resolve pipelines going to blocked state in AFLAC Prod Env
Description: Pipelines in AFLAC would go to Blocked state as the MongoDB service on the IWX Edge node crashes because of the compatibility issues of MongoDB...
Tue, 5 Oct, 2021 at 10:16 PM
How to remove duplicate records in full load ingestion?
Description: Set the below-advanced configuration to remove duplicate records during full load ingestion. This can be set to either the source or at table l...
Mon, 9 Aug, 2021 at 1:35 PM
How to map existing EMR/Dataproc cluster to Persistent compute
Infoworks 5.0 versions provides the ability to launched persistent cluster from Infoworks UI. However if one wants to map an existing cluster to infoworks p...
Wed, 11 Aug, 2021 at 5:07 PM
How to take backup of Infoworks Postgres DB
Description: Perform the below steps to take back up of Infoworks Postgres DB. i) source /opt/infoworks/bin/env.sh Assuming that Infoworks is instal...
Mon, 12 Jul, 2021 at 2:29 PM
How to debug SSL Handshake Exceptions and possible causes
Description: A TLS/SSL handshake failure occurs when a client and server cannot establish communication using the TLS/SSL protocol. When this error occurs i...
Mon, 21 Jun, 2021 at 7:01 PM
How to configure the Spark memory configurations for a interactive job of a Spark Pipeline
Description: Starting from IWX v3.1.2, we can provide the Spark configurations like spark driver memory, executor memory, and any other spark configurations...
Wed, 2 Jun, 2021 at 12:47 PM