Problem Description:
Ingestion is failing with below error for CSV source(source file location is sftp) when the target is gs bucket,
21/04/07 10:23:03 INFO io.infoworks.saas.ingestion.core.datacrawler.DistJobsDriver: Job finished
Exception in thread "main" java.lang.NoClassDefFoundError: com/jcraft/jsch/SftpProgressMonitor
at io.infoworks.saas.ingestion.core.connectors.file.datacrawler.AbstractFileDataCrawler.copyFilesFromSFTPServer(AbstractFileDataCrawler.java:174)
at io.infoworks.saas.ingestion.core.connectors.file.datacrawler.AbstractFileDataCrawler.getFileDetailsMap(AbstractFileDataCrawler.java:162)
at io.infoworks.connectors.files.csv.datacrawler.CSVDataCrawler.runJob(CSVDataCrawler.java:106)
at io.infoworks.saas.ingestion.core.datacrawler.Crawler.crawl(Crawler.java:58)
at io.infoworks.saas.ingestion.core.datacrawler.Crawler.runCrawlJob(Crawler.java:85)
at io.infoworks.saas.ingestion.core.datacrawler.DistJobsDriver.submitCrawlJob(DistJobsDriver.java:82)
at io.infoworks.saas.ingestion.core.datacrawler.DistJobsDriver.runJobs(DistJobsDriver.java:43)
at io.infoworks.saas.ingestion.core.datacrawler.DistJobsDriver.main(DistJobsDriver.java:26)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
Root cause:
This error comes if jsch-0.1.54.jar is missing in the ingestion classpath.
Solution:
This is a bug and we have fixed it in the future releases, as workaround we can copy jsch-0.1.54.jar that is present in /opt/infoworks/lib/ingestion/ to /opt/infoworks/lib/ingestion/connectors/csv/lib/.
Applicable IWX versions:
IWX 4.4.0.1