General

Placeholder for KB's articles

Ingest job fails at merge stage with Table or view not found error
Problem statement: Ingest job fails at merge state with following error message in the databricks log  20/08/18 18:46:43 ERROR DistJobsDriver: org.apach...
Fri, 1 Oct, 2021 at 11:41 AM
Kafka ingestion fails with error Caused by: java.lang.NoSuchMethodError: com.univocity.parsers.csv.CsvFormat.setDelimiter
Problem Description: Kafka streaming ingestion fails with below error Caused by: java.lang.NoSuchMethodError: com.univocity.parsers.csv.CsvFormat.se...
Mon, 24 Aug, 2020 at 9:42 PM
Steps to take backup of MongoDb data and restore the data from the backup file
Description: Below are the steps to take the backup of Infoworks DataFoundry metadata which is stored in MongoDB and to restore the same.  Note: Ensure...
Tue, 12 Jul, 2022 at 2:52 AM
Crawl metadata failing with "Response Row size exceeds 64K bytes and is incompatible with the Client software" error in a Teradata source.
Problem Description:   Metacrawl of a Teradata source failing with below error if the row size exceeds 64K bytes, [ERROR] 2020-07-28 10:40:00,512 [mai...
Sat, 4 Sep, 2021 at 1:24 AM
java.lang.NoClassDefFoundError: commons/exception/IWException wile doing Parquet source Metacrawl
Problem Description Metadata Crawl for a Parquet Source fails with the below error in the job log. Error: A JNI error has occurred, please check you...
Fri, 24 Jul, 2020 at 8:40 PM
Incremental ingestion fails with "cannot perform MERGE as multiple source rows matched and attempted to update the same target row in the Delta table" error at merge phase.
Problem Description Incremental ingestion fails at the merge phase if the CDC data has multiple records for the given natural key with the below error. ...
Fri, 17 Jul, 2020 at 12:47 PM
Sample Post Ingestion Hook Script
Description Post hook ingestion is a feature where you can provide bash scripts to be run after the ingestion job completes. The post hook runs in the s...
Mon, 13 Jul, 2020 at 12:02 PM
Script for deleting jobs in Databricks workspace for 1000 job limit/workspace
Problem Description Whenever we submit the Ingestion jobs, the Pipeline jobs or the Export Jobs from Infoworks, Infoworks runs these jobs on the Databri...
Fri, 9 Oct, 2020 at 9:02 PM
String value 'NULL' present in csv file will be converted to null data in hive
Problem Description: During the CSV file ingestion, if there as a string value 'NULL' in your data, it will be ingested as NULL data in the hive...
Tue, 22 Oct, 2019 at 12:52 PM
Steps to change the Ingestion job logs to debug mode
PROBLEM DESCRIPTION Please refer this article to enable the DEBUG logging mode in Infoworks ADE. STEPS TO CONFIGURE THE DEBUG MODE Sometimes, we...
Wed, 6 Feb, 2019 at 3:11 PM