Problem statement: When user runs a considerable number (around 50) of pipeline builds with hive execution engine in parallel, the user may find that some pipeline builds fail randomly with the below error message
java.lang.RuntimeException: java.io.FileNotFoundException: /etc/hadoop/conf.empty/mapred-site.xml (Too many open files)
Root cause and Solution: The hive jobs triggered by infoworks open conf files such mapred-site.xml, core-site.xml etc. When user runs lot of concurrent jobs, one can hit limit on number of open files that can be opened concurrently.
To resolve this issue, please increase the open files limit for infoworks-user, hive user and hadoop user by setting the below configurations in /etc/security/limits.conf in all master nodes.
infoworks-user hard nofile 100000 infoworks-user soft nofile 100000 infoworks-user hard nproc 100000 infoworks-user soft nproc 100000 hive hard nofile 100000 hive soft nofile 100000 hive hard nproc 100000 hive soft nproc 100000 hadoop hard nofile 100000 hadoop soft nofile 100000 hadoop hard nproc 100000 hadoop soft nproc 100000