vRealize Loginsight : How to remove faulty Master node

Remove the existing master node from the cluster

  • Log into each vRealize Log Insight node as root via SSH or Console.
  • Stop the Log insight service on all nodes:

service loginsight stop

Note: The following steps are not to be performed on the master node as it will be removed from the cluster.

  • Set the CASSANDRA_CONF environment variable by running this command on all nodes:

export CASSANDRA_CONF=/storage/core/loginsight/cidata/cassandra/config

  • Switch to the current working Cassandra directory on all nodes:

cd /usr/lib/loginsight/application/lib/apache-cassandra-*/bin

  • Start the Cassandra service on all nodes:

./cassandra -R

Note: During startup, this command produces approximately 88 lines of output ending with Node localhost/ state jump to normal. Press the enter key to return the shell prompt.

Note: If Cassandra fails to start with the error Unable to gossip with any seeds, open /storage/core/loginsight/cidata/cassandra/config/cassandra.yaml in a text editor and remove any instances of the old Master node address, then attempt to start Cassandra again.

  • On the worker node that is to be promoted, identify the Host ID for the current master node:

./nodetool status

  • Using the worker node that is to be promoted, remove the master node using the node’s Host ID:

./nodetool removenode HostID

Note: Replace HostID with the Host ID noted in step 6.

  • Stop the Cassandra service on all nodes:

./nodetool stopdaemon

Create a new loginsight-config file to promote a worker node

  • Create a new loginsight-config file to promote a worker node
  • Change to the /storage/core/loginsight/config/ directory on all nodes.
  • Create a copy of the latest loginsight-config file on all nodes:

Note: vRLI holds onto the last 3 configurations, in sequential order. Copy the loginsight-config file with the highest number and name the new loginsight-config file with the next sequential number.

Example: cp loginsight-config.xml#5 loginsight-config.xml#6

  • Open the new loginsight-config file created in step 2 in a text editor on all nodes:

Example:  vi loginsight-config.xml#6

  • Delete the lines associated to the old master node on all nodes:


<daemon host=”loginsight.vmware.com” port=”16520″ token=”e73406ea-728b-44fe-aabd-0131ed5b1648″>

<service-group name=”standalone” />


  • Rename the service-group name for the new master node to “standalone” on all nodes:


<daemon host=”″ port=”16520″ token=”8ede24c6-a533-422e-953e-c8df34ae6f8e”>

<service-group name=”standalone” />


  • Save and close the file on all nodes.
  • Start the loginsight service on all nodes:
  • service loginsight start

Add a Worker Node to a vRealize Log Insight Cluster


Restore log events from the old master node

  • Log into the old Master node as root via SSH or Console.
  • Stop the loginsight service on the old Master node:
  • service loginsight stop
  • Move the logs from the old Master node node to the new Master node:

scp -r /storage/core/loginsight/cidata/store newnode:/storage/core/loginsight/cidata

scp -r /storage/core/loginsight/cidata/store/*-*-*-*-* newnode:/storage/core/loginsight/cidata

Note: Replace newnode with the IP address or FQDN of the new vRealize Log Insight Master node.

  • Log into the new Master node as root via SSH or Console.
  • On the new Master node, ensure the loginsight service is stopped:

service loginsight stop

  • On the new Master node, import the logs:

for bucket in $(ls /storage/core/loginsight/cidata/store | grep -v ‘generation\|buckets\|strata_write.lock’); do echo y | /usr/lib/loginsight/application/sbin/bucket-index add $bucket –statuses archived; done


The Log Insight service MUST be stopped before running bucket-index.

The buckets will be market as archived immediately, or you can remove the parameter –statuses archived.

To check all buckets have been added correctly: /usr/lib/loginsight/application/sbin/bucket-index show

On the new Master node, start the loginsight service:

service loginsight start

Leave a Reply

Your email address will not be published. Required fields are marked *