
"Partial partition detected" typically occurs when there is an unexpected TCP communication issue. Thhe following may be displayed in the RabbitMQ admin console.
The main RabbitMQ log file (such as /var/log/rabbitmq/rabbit@server002.log on Linux) should contain something like this.
=WARNING REPORT==== 12-Nov-2020::21:02:06 ===
closing AMQP connection <0.23322.0> (10.15.257.11:6359 -> 10.86.115.42:5672 - Shovel shovel001, vhost: '/', user: 'john.doe'):
client unexpectedly closed TCP connection
=INFO REPORT==== 12-Nov-2020::21:02:06 ===
node rabbit@server1 down: connection_closed
=INFO REPORT==== 12-Nov-2020::21:02:06 ===
node rabbit@server1 up
=ERROR REPORT==== 12-Nov-2020::21:02:06 ===
Partial partition detected:
* We saw DOWN from rabbit@server1
* We can still see rabbit@server3 which can see rabbit@server1
We will therefore intentionally disconnect from rabbit@server3
The rabbitmqctl cluster_status command can be used to view the current nodes in the cluster. In this example, "nodes" shows that there are 3 nodes in the cluster, and "running_nodes" shows that only one of the nodes is "running".
~]# rabbitmqctl cluster_status
Cluster status of node rabbit@server2
[{nodes,[{disc,[rabbit@server1,rabbit@server2,rabbit@server3]}]},
{running_nodes,[rabbit@server2]},
{cluster_name,<<"rabbit@server1.example.com">>},
{partitions,[]},
{alarms,[{rabbit@server1,[]},{rabbit@server3,[]}]}]
The rabbitmqctl node_health_check command can be used.
~]# rabbitmqctl node_health_check
Timeout: 70.0 seconds
Checking health of node rabbit@server2
Health check passed
In this scenario, you can try to restart the node, and then reissue the rabbitmqctl cluster_status command to see if the node is detected as a "running node".
Did you find this article helpful?
If so, consider buying me a coffee over at