Problem Description


 Fetch new tables or recrawl metadata job gets aborted with following error message

Source refers to one or more tables that do not exist in metadata.


Root cause

One of the possibilities is that mongodb metadata for the desired source might have duplicate table ids.
The duplicates are caused when the table names have the special character '.' . Due to this duplicates, the above mentioned jobs get aborted.

Solution


Remove duplicates of table ids from mongodb metadata using the attached python script


steps:

 

  • Place remove_duplicate_tables.py in a specific location under ${IW_HOME}
    example: ${IW_HOME}/script_for_removing_duplicates/

  • Make infoworks user the owner of the file
  • chown infoworks-user:infoworks-user ${IW_HOME}/script_for_removing_duplicates/remove_duplicate_tables.py


  • Source env.sh using the following command. This steps needs to performed everytime before running the script.
    source ${IW_HOME}/bin/env.sh


  • Run the python script with source id as an argument. Pass the source id of the sources which has duplicate table ids in mongo document. (in other words where fetch newtables/metadata recrawl gets aborted)

    python ${IW_HOME}/script_for_removing_duplicates/remove_duplicate_tables.py <source id>

    example:

    python ${IW_HOME}/script_for_removing_duplicates/remove_duplicate_tables.py e133f987a166bddc6008c5ff

    Applicable Infoworks DataFoundry Versions:


    v2.4.x,v2.5.x,v2.6.x,v2.7.x,v2.8.x,2.9.x