Remove the existing master node from the cluster
- Log into each vRealize Log Insight node as root via SSH or Console.
- Stop the Log insight service on all nodes:
service loginsight stop
Note: The following steps are not to be performed on the master node as it will be removed from the cluster.
- Set the CASSANDRA_CONF environment variable by running this command on all nodes:
export CASSANDRA_CONF=/storage/core/loginsight/cidata/cassandra/config
- Switch to the current working Cassandra directory on all nodes:
cd /usr/lib/loginsight/application/lib/apache-cassandra-*/bin
- Start the Cassandra service on all nodes:
./cassandra -R
Note: During startup, this command produces approximately 88 lines of output ending with Node localhost/127.0.0.1 state jump to normal. Press the enter key to return the shell prompt.
Note: If Cassandra fails to start with the error Unable to gossip with any seeds, open /storage/core/loginsight/cidata/cassandra/config/cassandra.yaml in a text editor and remove any instances of the old Master node address, then attempt to start Cassandra again.
- On the worker node that is to be promoted, identify the Host ID for the current master node:
./nodetool status
- Using the worker node that is to be promoted, remove the master node using the node’s Host ID:
./nodetool removenode HostID
Note: Replace HostID with the Host ID noted in step 6.
- Stop the Cassandra service on all nodes:
./nodetool stopdaemon
Create a new loginsight-config file to promote a worker node
- Create a new loginsight-config file to promote a worker node
- Change to the /storage/core/loginsight/config/ directory on all nodes.
- Create a copy of the latest loginsight-config file on all nodes:
Note: vRLI holds onto the last 3 configurations, in sequential order. Copy the loginsight-config file with the highest number and name the new loginsight-config file with the next sequential number.
Example: cp loginsight-config.xml#5 loginsight-config.xml#6
- Open the new loginsight-config file created in step 2 in a text editor on all nodes:
Example: vi loginsight-config.xml#6
- Delete the lines associated to the old master node on all nodes:
Example:
<daemon host=”loginsight.vmware.com” port=”16520″ token=”e73406ea-728b-44fe-aabd-0131ed5b1648″>
<service-group name=”standalone” />
</daemon>
- Rename the service-group name for the new master node to “standalone” on all nodes:
Example:
<daemon host=”172.23.120.174″ port=”16520″ token=”8ede24c6-a533-422e-953e-c8df34ae6f8e”>
<service-group name=”standalone” />
</daemon>
- Save and close the file on all nodes.
- Start the loginsight service on all nodes:
- service loginsight start
Add a Worker Node to a vRealize Log Insight Cluster
Restore log events from the old master node
- Log into the old Master node as root via SSH or Console.
- Stop the loginsight service on the old Master node:
- service loginsight stop
- Move the logs from the old Master node node to the new Master node:
scp -r /storage/core/loginsight/cidata/store newnode:/storage/core/loginsight/cidata
scp -r /storage/core/loginsight/cidata/store/*-*-*-*-* newnode:/storage/core/loginsight/cidata
Note: Replace newnode with the IP address or FQDN of the new vRealize Log Insight Master node.
- Log into the new Master node as root via SSH or Console.
- On the new Master node, ensure the loginsight service is stopped:
service loginsight stop
- On the new Master node, import the logs:
for bucket in $(ls /storage/core/loginsight/cidata/store | grep -v ‘generation\|buckets\|strata_write.lock’); do echo y | /usr/lib/loginsight/application/sbin/bucket-index add $bucket –statuses archived; done
Notes:
The Log Insight service MUST be stopped before running bucket-index.
The buckets will be market as archived immediately, or you can remove the parameter –statuses archived.
To check all buckets have been added correctly: /usr/lib/loginsight/application/sbin/bucket-index show
On the new Master node, start the loginsight service:
service loginsight start
