Target Audience: Hadoop Admins / Infoworks Admins


Phoenix Export fails with "Bulk load aborted with some files not yet loaded" message in logs


Possible error message in log looks like the below 


 INFO mapreduce.LoadIncrementalHFiles: Split occured while grouping HFiles, retry attempt 10 with 40 files remaining to group or split
 INFO hdfs.DFSClient: Cancelling HDFS_DELEGATION_TOKEN token 14706 for infoworks on hdfs://iw/data/information/
 ERROR mapreduce.LoadIncrementalHFiles: -------------------------------------------------
Bulk load aborted with some files not yet loaded:


and 


WARN mapreduce.LoadIncrementalHFiles: Attempt to bulk load region containing \x0A\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 into table EAGLE_20180908_103202:TABLE_NAME with files [family:AUDIT path:hdfs://iw/data/information//hfiles/DATA_2018_20173718.INFO_TABLE/AUDIT/772dbkakbskbdbbsy3ns, family:0 path:hdfs://iw/data.information/hfiles/DATA_2018_20173718.INFO_TABLE/AUDIT/TT30dhj8fn2nfhwbr92br89vvfbn2] failed


Explanation and debugging the above issues:


Whenever the Infoworks User runs the Phoenix export job, Infoworks first creates the temp location

as shown below


export to /iw/data/information/target/INFO_TABLE/hfiles 

and load the data into those temp file [ all the data ill be in hfile format]


Loading HFiles for DATA_2018_24234:INFO_TABLE

 from /iw/data.information/target/INFO_TABLE/hfiles/DATA_2018_24234.INFO_TABLE


After the above steps, the data will be moved into HBase staging directory which will consume the data into HBase tables.


If you see the below error [ or same sort of]


path:hdfs://iw/data.information/target/INFO_TABLE/hfiles/DATA_2018_24234.INFO_TABLE/AUDIT/772dbkakbskbdbbsy3ns,path:hdfs:///iw/data.information/target/INFO_TABLE/DATA_2018_24234.INFO_TABLE/0/772dbkakbskbdbbsy3ns] failed


Then you might need to configure the hbase.bulkload.staging.dir settings in HBase.


The user should set the above settings in Hbase and make user should have all privileges to perform the operations. Once the required settings/privileges are provided, please re-run the export, This should work fine.


Thanks,

Sri