Bootstrap FreeKB - RabbitMQ - Resolve "Partial partition detected"
RabbitMQ - Resolve "Partial partition detected"

Updated:   |  RabbitMQ articles

"Partial partition detected" typically occurs when there is an unexpected TCP communication issue. Thhe following may be displayed in the RabbitMQ admin console.

 

The main RabbitMQ log file (such as /var/log/rabbitmq/rabbit@server002.log on Linux) should contain something like this.

=WARNING REPORT==== 12-Nov-2020::21:02:06 ===
closing AMQP connection <0.23322.0> (10.15.257.11:6359 -> 10.86.115.42:5672 - Shovel shovel001, vhost: '/', user: 'john.doe'):
client unexpectedly closed TCP connection

=INFO REPORT==== 12-Nov-2020::21:02:06 ===
node rabbit@server1 down: connection_closed

=INFO REPORT==== 12-Nov-2020::21:02:06 ===
node rabbit@server1 up

=ERROR REPORT==== 12-Nov-2020::21:02:06 ===
Partial partition detected:
* We saw DOWN from rabbit@server1
* We can still see rabbit@server3 which can see rabbit@server1
We will therefore intentionally disconnect from rabbit@server3

 

The rabbitmqctl cluster_status command can be used to view the current nodes in the cluster. In this example, "nodes" shows that there are 3 nodes in the cluster, and "running_nodes" shows that only one of the nodes is "running".

~]# rabbitmqctl cluster_status
Cluster status of node rabbit@server2
[{nodes,[{disc,[rabbit@server1,rabbit@server2,rabbit@server3]}]},
 {running_nodes,[rabbit@server2]},
 {cluster_name,<<"rabbit@server1.example.com">>},
 {partitions,[]},
 {alarms,[{rabbit@server1,[]},{rabbit@server3,[]}]}]

 

The rabbitmqctl node_health_check command can be used.

~]# rabbitmqctl node_health_check
Timeout: 70.0 seconds
Checking health of node rabbit@server2
Health check passed

 

In this scenario, you can try to restart the node, and then reissue the rabbitmqctl cluster_status command to see if the node is detected as a "running node".

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee



Comments


Add a Comment


Please enter b7bccd in the box below so that we can be sure you are a human.