Problem Statement: Any new triggered workflow is stuck in a pending state and does not move to run state even after a considerable time. 



Possible root causes and resolutions: 

  1. Root cause: Orchestrator scheduler or Orchestrator Worker services are in a stopped state.
    Resolution: Check the status of the services and start the orchestrator if it is down.
    Use the below commands for the same.
    {IW_HOME}/bin/status.sh
{IW_HOME}/bin/start.sh orchestrator


  2. Root cause: Reached the limit on maximum workflow tasks

    Every infoworks deployment comes with a default, but configurable, limit on a maximum number of tasks (workflow tasks) that can be run parallelly across the whole deployment.

    This limit can be seen using the following command (can also be configured in the same file):

    infoworks@iw-azure-cs-dbc-vm:~$ cat /opt/infoworks/orchestrator-engine/airflow.cfg | grep parallelism
parallelism = 32

    This means only 32 workflow tasks can be run in parallel across all the workflows within the infoworks deployment. While you have 32 tasks running, any new triggered workflow will go into a pending state.
    Please note this value can be different in your environment. We will use 32 as examples for the rest of the article.

    Resolution:
    Firstly, one can check the number of current running task instances from the airflow ui page which can be accessible at http://edge_node_ip:3003/


    Note: One needs to navigate and add a filter as shown in the above picture.

    Now one can wait for this number to come down or cancel the workflows occupying all the tasks.

    How to cancel these workflows?
    Please see the attached video for the same.

    Ideally, after canceling few workflows, you should see the triggered workflows going into running state.

    Applicable Infoworks Versions:

    IWX  v3.x,3.1.x,v4.x