Installing HA with 3 node setup for Infoworks


Applicable on: Infoworks Version- 2.3.X, 2.4.X, 2.5.X, 2.6.X


Requirements :


3 Edge Node:[ 1 Active, 1 passive and 1 HA]


We also, assume that end user has access to all the 3 edge nodes as a infoworks user and has rights to generate ssh key, has a home directory(/home/infoworks/) and also to /opt/  path.


  1. Login to 1st machine(active machine) and perform the below mentioned steps to complete the prerequisites for infoworks and HA enablement

cd /home/[infoworksuser]/

ssh-keygen

cd /home/[infoworksuser]/.ssh/

touch authorized_keys

chmod 600 authorized_keys



Download and extract the tarball to the dedicated folder, for example, /opt.

Untar the tarball. This creates a folder named infoworks inside /opt.


sudo tar -xvf <tar file> -C /opt


Create a dedicated system user called <Infoworks_user>


NOTE: 

The <Infoworks_user> must belong to a group that is present in hadoop.proxyuser.hive.groups. This will allow hive to perform actions on behalf of <Infoworks_user>.


Ensure that the Infoworks folder is owned by <Infoworks_user> and start all Infoworks components by running the following start.sh script in infoworks/bin folder as <Infoworks_user> with the argument all.


Python-Pip Packages needs to be upgraded to make all the components work properly in Edge node. Please perform the below steps in Active node’s home directory


cd  $IW_HOME

source env.sh

pip install --upgrade pip



sudo chown -R <Infoworks_user>:<Infoworks_user> /opt/infoworks

cd /opt/infoworks/bin

sudo -u <Infoworks_user> ./start.sh all



Users can check the status of the running components using the below command

cd $IW_HOME/bin

source env.sh

./status.sh



After validating the status we need to stop all the services using the below commands


./stop.sh all mongo



After the above steps validate once that all the services including mongo are stopped


  2.   Login to second machine/Edgenode( Passive machines) and  perform the below mentioned steps to complete the prerequisites for infoworks and HA enablement


cd /home/[infoworksuser]/

ssh-keygen

cd /home/[infoworksuser]/.ssh/

touch authorized_keys

chmod 600 authorized_keys



Download and extract the tarball to the dedicated folder, for example, /opt.

Untar the tarball. This creates a folder named infoworks inside /opt.


sudo tar -xvf <tar file> -C /opt


Create a dedicated system user called <Infoworks_user>



NOTE: 

The <Infoworks_user> must belong to a group that is present in hadoop.proxyuser.hive.groups. This will allow hive to perform actions on behalf of <Infoworks_user>.


Ensure that the Infoworks folder is owned by <Infoworks_user> and start all Infoworks components by running the following start.sh script in infoworks/bin folder as <Infoworks_user> with the argument all.


Python-Pip Packages needs to be upgraded to make all the components work properly in Edge node. Please perform the below steps in Passive node home directory

cd  $IW_HOME

source env.sh

pip install --upgrade pip

sudo chown -R <Infoworks_user>:<Infoworks_user> /opt/infoworks

cd /opt/infoworks/bin

sudo -u <Infoworks_user> ./start.sh all



Users can check the status of the running components using the below command


cd $IW_HOME/bin

source env.sh

./status.sh



After validating the status we need to stop all the services using the below commands


./stop.sh all mongo


After the above steps validate once that all the services including mongo are stopped


3. Login to 3rd Machine/edge node (HA Machine) and follow the below steps


Download and extract the tarball to the dedicated folder, for example, /opt.

Untar the tarball. This creates a folder named infoworks inside /opt.


sudo tar -xvf <tar file> -C /opt


Create a dedicated system user called <Infoworks_user>

NOTE: Please don’t start any services like you did in earlier machines



Ensure that the Infoworks folder is owned by <Infoworks_user> and start all Infoworks components by running the following start.sh script in infoworks/bin folder as <Infoworks_user> with the argument all.


sudo chown -R <Infoworks_user>:<Infoworks_user> /opt/infoworks

cd /opt/infoworks/bin



Confirm that you are in $IW_HOME/bin folder and source env.sh file


source env.sh



Python-Pip Packages needs to be upgraded to make all the components work properly in Edge node. Please perform the below steps in Active node’s home directory


cd  $IW_HOME

source env.sh

pip install --upgrade pip




In the bin directory ($IW_HOME/bin) Download the HA script using the below command


cd $IW_HOME/bin

wget https://s3.amazonaws.com/infoworks-resources/tmp/scripts/install-ha.sh -O install-ha.sh  



This script will create a HA directory inside $IW_HOME/ directory 


Execute the below command in $IW_HOME/bin/ path


cd $IW_HOME/bin

bash install-ha.sh



After the execution gets completed please navigate to the below directory and make copy of the file as shown

cd $IW_HOME/conf/

cp ha.conf.default  ha.conf


We have made duplicate file of ha.conf.default as ha.conf now edit that file



vi ha.conf


Modify the values as mentioned below

change the host2_iw_home value from /apps/infoworks  to /opt/infoworks


host2_iw_home=/apps/infoworks to host2_iw_home=/opt/infoworks



After making the modifications, please save the file and exit from vi editor.


update $IW_HOME/conf/ha.conf. See below steps for configuring the same.

 

The HA configurations are stored in IW_HOME/conf/ha.conf, please edit as required to modify the settings as highlighted below

 

Following shows a sample configuration of HA:


# hostname of the server, string
# Each server should be accessible by the other using the hostname
host1_hostname=1st_MACHINE’s IP ADDRESS
# type - local/remote
# Local implies the installation is on the same node as the HA tool
# Remote implies the installation is on a different node but accessible via ssh
host1_type=active
# role of the installation - active/passive
# During failover, the HA tool automatically modifies this value
host1_role=active
# Infoworks services are running on host1 in this case so set the role to active
#SSH login details
host1_ssh_keyless=true
host1_ssh_private_key=
host1_username=infoworks
# IW_HOME value for this host
host1_iw_home=/opt/infoworks
host2_hostname=2nd_machines’IP ADDRESS
host2_type=remote
host2_role=passive
# Node on which HA is instlled in this case.
host2_ssh_keyless=false
host2_ssh_private_key=/home/infoworks/.ssh/id_rsa
host2_iw_home=/opt/infoworks
host2_username=infoworks
# During failover, if all services on the failing active node do not stop,
# The failover is halted and the following script will be executed
# The output the of the script will stored in the configured log folder
failover_error_handler_script=/path/to/script.sh
failover_error_handler_log=/path/to/log
# If any service is not used and is stopped at all times, HA can be configured to ignore this service
# comma separated string of Component Names (as shown in output status.sh), spaces are acceptable
services_error_whitelist=

#############################################################




Navigate to /home/infowowks_user home folder
Generate the ssh key

Navigate to infoworks user’s home directory 

cd
ssh-keygen


The generated keys will be in infoworks users directory as shown below

/home/[infoworksuser]/.ssh/


4. Copy-paste the id_rsa content from 3rd machine(HA machine) to 1st machine (Active) and 2nd Machine(Passive machine) to the 1st machine's authorized_keys file located /home/[infoworksuser]/.ssh/ path.

Please ensure to enter the 2nd machine's Id_rsa in next line after entering the HA(3rd) machines id_rsa in authorized_keys file. Both the keys from 3rd and 2nd machines must be present in 1st machine's authorized_keys file one after the other.


In HA Machine(3rd machine) navigate to /home/[infoworksuser]/.ssh/ folder and cat the content of id_rsa file and copy the content correctly


cat /home/[infoworksuser]/.ssh/id_rsa


Copy the output of the above command and login to 1st machine(Active Machine) paste the content into /home/[infoworksuser]/.ssh/authorized_keys file and save it


After the above step please copy the id_rsa content into 2nd machine (passive machine) file located in /home/[infoworksuser]/.ssh/authorized_keys file and save it


After the above steps, try logging into 1st Machine(Active Machine) from 3rd Machine(HA Machine) using ssh command


ssh infoworksuser@ip_address_of_1st_machine


 

After successful login , please logout and repeat the ssh login for 2nd machine from HA(3rd machine) as as shown below

 

ssh infoworksuser@ip_address_of_2nd_machine



It should work fine and user should be able to login to 1st machine from 3rd machine successfully and user should be able to login to 2nd machine from 3rd machine successfully


After verifying the ssh, please logout from ssh connection and get back to 3rd machines.


4. Login to 1st Machine(Active machine) as a infoworks user and perform the below steps


cd $IW_HOME/bin
source env.sh
sh start-active.sh



5. Login to 2nd Machine(Passive machine) as a infoworks user and perform the below steps


cd $IW_HOME/bin
source env.sh
sh start-passive.sh  ip_address_of_1st_machine



Note: above command takes 1 parameter and it should be the ip address of Active machine(1st machine)
and
After the above steps the only mongo should be running, user can check the status by running ./status.sh command

 

6.  Login to 1st Machine(Active machine) as a infoworks user and perform the below steps



cd $IW_HOME/bin
source env.sh
cd $IW_HOME/resources/mongodb/conf/
vi mongodb.conf



[make the below changes in the file]

Set #noauth=true  to noauth=true [ remove comment for this line]
Set auth=true  to #auth=true [Add the comment to this line]


Save the file and exit from the vi editor


Navigate to $IW_HOME/bin folder

cd $IW_HOME/bin
source env.sh
./stop.sh mongo && ./start.sh mongo



After the above steps if required you can validate the mongo status using ./status.sh command


After the above steps user needs to add a iw-ha user to the MongoDb and create a role clusteradmin for the same user. The steps to perform to make this happen are listed below.


clear
mongo



[user will be logged into mongodb console after running the above command]


use admin




[above command is used to choose the admin database]


db.createUser({user: "iw-ha", pwd: "IN11**rk", roles: [ "clusterAdmin" ]});


exit;




[above command will exit you from Mongodb console]

Now we need to reverse the authentication setting in mongodb.conf file.


cd $IW_HOME/bin
source env.sh
cd $IW_HOME/resources/mongodb/conf/
vi mongodb.conf



[make the below changes in the file]

Set noauth=true  to #noauth=true [ remove comment for this line]
Set #auth=true  to auth=true [Add the comment to this line]



Save the file and exit from the vi editor

 


Navigate to $IW_HOME/bin folder
cd $IW_HOME/bin
source env.sh


After the above steps we need to enable the mongoKey value in 1st machine(Active Machine)


Navigate to $IW_HOME/conf and perform the steps



cd $IW_HOME/conf
vi mongoKey
[add the below key in mongoKey file and save it]
RVSK4PAER28AUVEIVNA#BLYK
chmod 400 mongoKey


After the above settings please navigate to $IW_HOME/resources//mongodb/conf/ directory

cd $IW_HOME/resources//mongodb/conf/
vi mongodb.conf
[Add the below line in the mongodb.conf file]
keyFile=$IW_HOME/conf/mongoKey




Save the file and exit from vi editor


Navigate to $IW_HOME/bin


source env.sh
./stop.sh mongo && ./start.sh mongo




7. Login to 2nd Machine(passive machine) as a infoworks user and perform the MongoKey setup in Passive machine as mentioned below

Navigate to $IW_HOME/conf and perform the steps



cd $IW_HOME/conf
vi mongoKey
[add the below key in mongoKey file and save it]
RVSK4PAER28AUVEIVNA#BLYK
chmod 400 mongoKey


After the above settings please navigate to $IW_HOME/resources//mongodb/conf/ directory

cd $IW_HOME/resources//mongodb/conf/
vi mongodb.conf




[Add the below line in the mongodb.conf file]

keyFile=$IW_HOME/conf/mongoKey



Save the file and exit from vi editorand navigate to $IW_HOME/bin directory

cd $IW_HOME/bin
source env.sh
./stop.sh mongo && ./start.sh mongo




8. Login to 1st machine(Active machine ) as Infoworks user and perform the below steps


cd $IW_HOME/bin
Source env.sh
./start-active.sh



9. Login to 2nd machine(Passive Machine) as Infoworks user and perform the below steps


cd $IW_HOME/bin
Source env.sh
./start-passive [1st_machine_ip]



10. Login to 3rd Machine(HA Machine) as Infoworks user and perform the below steps


cd $IW_HOME/conf
touch configured
cp conf.properties.default conf.properties



edit conf.properties in vi, and run the below command after hitting escape+shit +:

:
%s/{{INFOWORKS_HOME}}/\/opt\/infoworks/g



After replacing the string using above command, perform the below steps


cd $IW_HOME/bin
vi start.sh



[please comment the below mentioned lines(first 3 lines and last line)

 

#if [ ${is_process_running} == true ]; then
#  echo "Process like HA Service is already running"
#else
echo -e "Starting HA Service" | tee -a ${LOG_FILE} INFOWORKS_LOGS_BASE_PATH=`grep -v '^#' ${CONF_PROPS}| grep -o 'iw_logs_base_dir=.*' | cut -d '=' -f 2`
      HA_LOGS_DIR=${INFOWORKS_LOGS_BASE_PATH}/ha
  mkdir -p ${HA_LOGS_DIR}
HA_OUT_FILE=${HA_LOGS_DIR}/ha.out
if [ -f "${HA_OUT_FILE}" ]; then
NOW=$(date "+%F-%T")
mv "${HA_OUT_FILE}" "${HA_OUT_FILE}-${NOW}"
Fi
export IW_HA_CONF=${IW_HOME}/conf/ha.conf
export IW_HA_LOG=${HA_LOGS_DIR}/ha.log
 export IW_MONGO_CONF=${IW_HOME}/resources/mongodb/conf/mongodb.conf
export IW_MONGO_HOME=${IW_HOME}/resources/mongodb
pushd ${IW_HOME}/HA >> ${LOG_FILE}
nohup python2.7 main.py &> ${HA_OUT_FILE} & echo $! > ${IW_HOME}/HA/ha.pid
echo $! > ha_pid
popd >> ${LOG_FILE
#fi



After making changes, please save the file and exit from vi editor.


Navigate to $IW_HOME/bin


cd $IW_HOME/bin
Source env.sh
./start.sh ha



After the above settings in 3rd machine(HA machine) only HA should be running.

User can validate the running process by running the below command

cd $IW_HOME/bin
./status.sh


User can see the logs for HA component in $IW_HOME/logs/ha/ha.log