Root Cause
When writing data to S3 during a multi-part upload , the maximum number of parts can be 10000
Due to this threshold the jobs fail during execution
Solution
To overcome this limitation we need to increase the size of the ‘part’ with the below advanced configuration.
The part size can be up to 5 GB
Set the below property as an additional configuration in table properties as shown in below image. This will increase the capacity for each 'part'
key: ingestion_spark_configs
value: spark.hadoop.fs.s3a.multipart.size=104857600
Please rerun the job after setting the above advanced configuration.
Affects Version:
IWX 3.2,4.2 and above
