Problem Description:

In a source with the target hdfs path pointing to s3 filesystem ex: s3://bucket/folder, Infoworks truncate operation is not working though the status is shown as successful in the UI. 

messages from the truncate log look like below,

[INFO] 2021-03-03 11:53:28,232 [main] infoworks.tools.purge.PurgeUtil:653 :: Deleting hdfs path : s3://customer-success-test-bucket/test_s3/greentrip/
[INFO] 2021-03-03 11:53:29,536 [main] infoworks.tools.purge.PurgeHDFS:31 :: Unable to Delete Directory s3://customer-success-test-bucket/test_s3/greentrip


Solution:

This is a code bug and we can resolve this by following below mentioned steps,

    1. Log in to the IWX Edge node as an Infoworks user.
    2. Stop all Infoworks jobs if any.
    3. cd /opt/infoworks/bin/
    4. source env.sh
    5. mv tools.jar tools.jar.backup
    6. wget https://infoworks-resources.s3.amazonaws.com/aflac/tool-jar/tools.jar
    7. cat /opt/infoworks/conf/conf.properties | grep iw_hdfs_temp_home (This will return you the temp location on hdfs)
    8. hadoop fs -rm -r <location from previous step>/iwjobs/libjars/opt/
infoworks/bin/* (This should move tools.jar from cache to trash)


Applicable IWX versions:

IWX 3.1.2-emr