Pipeline Job failing with Dates older than '1582-10-15' or Timestamp older than ‘1900-01-01 00:00:00.0' : Infoworks

Pipeline Job failing with Dates older than '1582-10-15' or Timestamp older than ‘1900-01-01 00:00:00.0' Print

Modified on: Fri, 22 Sep, 2023 at 9:19 AM

This is due to Spark 2.X's legacy calendar.

Please add the below Spark configuration to your pipeline job and retry the job

Key:iw_spark_app_conf
Value: spark.sql.legacy.parquet.int96RebaseModeInWrite=LEGACY;spark.sql.legacy.parquet.datetimeRebaseModeInWrite=LEGACY

Did you find it helpful? Yes No