Problem statement: When user runs a considerable number (around 50) of pipeline builds with hive execution engine in parallel, the user may find that some pipeline builds fail randomly with the below error message 

java.lang.RuntimeException: /etc/hadoop/conf.empty/mapred-site.xml (Too many open files)

Root cause and Solution: The hive jobs triggered by infoworks open conf files such mapred-site.xml, core-site.xml etc. When user runs lot of concurrent jobs, one can hit limit on number of open files that can be opened concurrently.

To resolve this issue, please increase the open files limit for infoworks-user, hive user and hadoop user by setting the below configurations in /etc/security/limits.conf in all master nodes. 

infoworks-user hard nofile 100000
infoworks-user soft nofile 100000
infoworks-user hard nproc 100000
infoworks-user soft nproc 100000

hive hard nofile 100000
hive soft nofile 100000
hive hard nproc 100000
hive soft nproc 100000

hadoop hard nofile 100000
hadoop soft nofile 100000
hadoop hard nproc 100000
hadoop soft nproc 100000