Root Cause

When writing data to S3 during a multi-part upload , the maximum number of parts can be 10000

Due to this threshold the jobs fail during execution


To overcome this limitation we need to increase the size of the ‘part’ with the below advanced configuration. 

The part size can be up to 5 GB 

Set the below property as an additional configuration in table properties as shown in below image. This will increase the capacity for each 'part'

key: ingestion_spark_configs

value: spark.hadoop.fs.s3a.multipart.size=104857600

Please rerun the job after setting the above advanced configuration.

Affects Version:

IWX 3.2,4.2 and above