High Availability ( HA ) INSTALLATION STEPS

1) Have 7 VMs ready - 3 databases,  2 appservers/webservers,  2 mediation servers

2) Configuring Databases

before start db, may want to remove auto update
sudo apt remove unattended-upgrades

2a set IPs in 3 database servers and reboot
e.g. 
10.255.20.125 (keep hostname same as mysqldb1)
10.255.20.126 (change hostname to mysqldb2)
10.255.20.127 (change hostname to mysqldb3)

2b after reboot, update /etc/hosts and make sure you can ping each hostname
10.255.20.125 mysqldb1
10.255.20.126 mysqldb2
10.255.20.127 mysqldb3

2c sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf and make sure following entries are correct
max_connections = 500
innodb_buffer_pool_size=2G

#only on database server2
[mysqld]
server_id = 2

#only on database server3
[mysqld]
server_id = 3

note: optionally, you can take a snapshot at this point
on mysqldb1
sudo systemctl restart mysql

2d
change uudi of mysqldb2 and mysqldb3
sudo systemctl stop mysql
sudo rm /var/lib/mysql/auto.cnf
sudo systemctl start mysql

2e
create DBHA instance
on mysqldb1
mysqlsh mysqldb1 -u clusteradmin
mysql-py []> dba.configure_local_instance('clusteradmin@mysqldb1')

on mysqldb2
mysqlsh mysqldb2 -u clusteradmin
mysql-py []> dba.configure_local_instance('clusteradmin@mysqldb2')

on mysqldb3
mysqlsh mysqldb3 -u clusteradmin
mysql-py []> dba.configure_local_instance('clusteradmin@mysqldb3')

2f
on mysqldb1
dba.create_cluster('DBCluster');
cluster=dba.get_cluster('DBCluster')

cluster.add_instance('clusteradmin@mysqldb2')
cluster.add_instance('clusteradmin@mysqldb3')

note: select c for clone or enter
cluster.status()

3.  Configuring Appservers
3a 
edit /etc/hosts to include mysqldb1, mysqldb2, mysqldb3
validate appservers can ping all 3 dbhost

3b edit jgroups-channelfactory-stacks.xml file located  in /opt/VAroot/cruzoc/oware/jboss-5.1/server/oware/deploy/cluster/jgroups-channelfactory.sar/META-INF/ directory

search initial_hosts and replace IP

3c edit synergy-override to comment out 127.0.0.1
  edit portal_ext to include port 6446 (may already be there)
/opt/VAroot/cruzoc/oware/synergy/tomcat-7.0.70/webapps/ROOT/WEB-INF/classes

3d
sudo mysqlrouter --bootstrap clusteradmin@10.255.20.125:3306  --conf-use-sockets --account routerfriend --account-create always --user=root
may need to change to for 6446 to auto bind after reboot
on app1
sudo mysqlrouter --bootstrap clusteradmin@10.255.20.125:3306  --directory /home/synergy/mysqlrouter --conf-use-sockets --account routerfriend --user=synergy --force --account-create always --https-port=9443

sudo mysqlrouter --bootstrap clusteradmin@10.255.20.116:3306  --directory /home/synergy/mysqlrouter --conf-use-sockets --account routerfriend --user=synergy --force --account-create always --https-port=9443

on app2
sudo mysqlrouter --bootstrap clusteradmin@10.255.20.125:3306  --directory /home/synergy/mysqlrouter --conf-use-sockets --account routerfriend --user=synergy --force --https-port=9443
sudo mysqlrouter --bootstrap clusteradmin@10.255.20.116:3306  --directory /home/synergy/mysqlrouter --conf-use-sockets --account routerfriend --user=synergy --force --https-port=9443

reboot the apps vm
sudo reboot

to confirm port 6446 is listening execute:
netstat -na | grep 6446
netstat -na | grep 9443

3e
synergy@synergy:~$ pingdb -u clusteradmin -p synergy\P\@\$\$

pingdb started...

Loading installed.properties: /opt/VAroot/cruzoc/owareapps/installprops/lib/installed.properties

Loading owdatabase.properties: /opt/VAroot/cruzoc/oware/lib/owdatabase.properties                                                  
DBMS Environment
       Host:  10.255.20.121
       Port:  6446
       Name:  owbusdb
       User:  clusteradmin

pingdb completed successfully.

loaddb -u clusteradmin -w synergy\P\@\$\$

loaddb -u clusteradmin -w synergy\P\@\$\$ -s

3f
go to primary db and 
on db server = 10.255.20.125
     copy pmAddPrimaryKeys.sql & taAddPrimaryKeys.sql 

mysql -u clusteradmin -h mysqldb1 -p owbusdb < taAddPrimaryKeys.sql
mysql -u clusteradmin -h mysqldb1 -p owbusdb < pmAddPrimaryKeyNoSeq.sql

note: (may sure pm script includes alter pm_ret_table_seq);
will need to run again after appserver starts
mysql -u clusteradmin -h mysqldb1 -p owbusdb < pmAddPrimaryKey.sql

3g
on app1 10.255.20.121
execute dbpostinstall

3h
on app1 import cluster license (licenseimporter licensefile.xml <ENTER>)

sudo service oware start
when ready
sudo service synergy start

alter table owbusdb.pm_ret_table_seq change id id int not null primary key;
(may not needed)
check the table in primary db
mysql> describe table owbusdb.pm_ret_table_seq;

on app2 
sudo service oware start
when ready
sudo service synergy start

Troubleshooting: If web has issue connect to app
1) delete oware/temp directory
2) check installed.properties mulitcast and config server IP,  shall be match app1 IP
3) portal ext,  oware.appserver.ip=10.255.20.121 for web1,  10.255.20.122 for web2,  same for synergy.virtual address , when without load blancer

4. Configuring Medservers
4a change IP 
4b on installed.properties 
change partition name if more than 1 partition  
change app IP
change config server and  multicast IP

4c change med IP in stack file
can search for string 192
/opt/VAroot/cruzoc/oware/jboss-5.1/server/oware/deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml

login to control panel add
cruzmed1-medPartition

upgrade from 10.0.5 to 10.0.7
tar -xvf CruzOC_DIST_10.0.7.38.tar

change app install property to port 6446
change web install property to port 6446

sudo service synergy stop,  if not stopped, kill the process