Infoworks export to postgres database uses spark execution engine. The properties file used for launching these spark jobs is present at /opt/infoworks/conf/spark-export.conf.

Below is the list of configurations needed for successful Postgres export but are often missed. 

  1. One may encounter export failures while yarn getting launched due to incorrect classpath (ex: java.lang.NoSuchMethodError, java.lang.ClassNotFoundException etc). One can then replace the iw_export_classpath with the below one in /opt/infoworks/conf/conf.properties
    iw_export_classpath=/opt/infoworks/lib/extras/export/*:/usr/lib/spark/jars/*:/opt/infoworks/lib/mongodb/mongo-java-driver-3.8.0.jar:/opt/infoworks/bin/tools.jar:/opt/infoworks/lib/exec/commons-exec-1.2.jar:/opt/infoworks/lib/jackson/*:/opt/infoworks/lib/shared/*:/opt/infoworks/lib/commons-lang3/*:/usr/lib/spark/conf/:/usr/lib/hive/conf/:/usr/lib/hive/lib/*:/usr/lib/hive-hcatalog/share/hcatalog/*:/opt/infoworks/platform/bin/notification-common.jar:/opt/infoworks/platform/bin/platform-common.jar:/opt/infoworks/platform/lib/notification-client/*:/opt/infoworks/lib/ingestion/*


  2. The properties file /opt/infoworks/conf/spark-export.conf should include the following properties 

    spark.master yarn
    spark.driver.extraJavaOptions -Dhdp.version=<hdp-version-number>
    spark.yarn.am.extraJavaOptions -Dhdp.version=<hdp-version-number>
    spark.sql.hive.convertMetastoreOrc false
    spark.sql.hive.convertMetastoreParquet false


    Note: If hdp-version number is not applicable to the environment one is using (ex: EMR), please the set <hdp-version-number> as 2.6.4.0-91
    i. e. the property will look like spark.yarn.am.extraJavaOptions -Dhdp.version=2.6.4.0-91

     


    Applicable versions: 3.1x