Steps to troubleshoot if your alarms/events stop being received by the CruzOC server
Delete queue files & oware/temp Directory
Check to see if there is a queue file under $OWARE_USER_ROOT/owareapps/eventmgmt/temp in the Medservers and also the App servers.
If it does, please try the following:
1) Shutdown Appservers and Medservers
2) Delete the queue files
3) Delete oware/temp
4) Restart Appservers and Medservers
If this does not resolve your issue please Enable the following Trace-Level Debug
Enable Trace Level Debug
Turn on some trace-level debug for the following category:
com.dorado.core.mediation.jmx.OWHeartBeat
If you can modify the existing category in ../owareapps/ezmediation/conf/ezmediation-log4j.xml from INFO to TRACE should look like this:
<category name="com.dorado.core.mediation.jmx.OWHeartBeat">
<priority value="TRACE"/>
</category>
Then restart the mediation server. That will give us more information around the med server/app server connection issue.
The OWHeartBeat class makes a call to the application server, invoking the method agentHeartbeat(OWAssignmentBean agent) on the OWMedServerTrackerMBean, against all servers in the application server cluster.
In addition to the trace-level logging on the med server, turn on trace-level logging on the application server too.
Here is the category (should already be in the .xml file, you just need to change the priority)
<category name="com.dorado.mbeans.OWMedServerTrackerMBean">
<priority value="TRACE"/>
</category>
Once you've collected 15-20 minutes of logging on both servers, you should have enough data to analyze to see how the application server is responding to the mediation server.
During that period, also collect Wireshark captures, perhaps on both the servers, to see if the application server is actually getting all the heartbeat calls and then not responding to half of them or if they are only getting half the attempts.
Collect the Getlogs files and send them to support for analysis.
Run Getlogs Command
Source OWARE environment (Windows Only)
cmd line > oware for windows
cmd line > getlogs
The last line of the script will provide you the location of the logs.jar file.
Rename the logs.jar file to HD-TicketNumberLogs.jar then send log files to https://wetransfer.com addressing: tsc@doradosoftware.com
Creator: Ed/TV 6/11/2020
Verified: