Problem Description 


Infoworks mongo metadata database runs in a three node master slave setup when HA is enabled. If service on one node is down for an extended period of time replication goes out of sync. and rs.status() shows the slave struck in recovering state


Root cause


Default oplog sizes are insufficient to handle extended periods of node unavailability.


Solution


After getting connected to mongo shell, command rs.printReplicationInfo() gives the current oplog size. Please refer to the following tutorial to set a larger oplog size. We recommend setting this to >= 5GB (depends on transactional volume and anticipated downtime interval). 


https://docs.mongodb.com/manual/tutorial/change-oplog-size/


Applicable Infoworks versions 


Any version subsequent to Infoworks Datafoundry >v2.8 (2.8, 2.9, 3.1)