RabbitMQ - Resolve "Partial partition detected"

"Partial partition detected" typically occurs when there is an unexpected TCP communication issue. Thhe following may be displayed in the RabbitMQ admin console.


The main RabbitMQ log file (such as /var/log/rabbitmq/rabbit@server002.log on Linux) should contain something like this.

=WARNING REPORT==== 12-Nov-2020::21:02:06 ===
closing AMQP connection <0.23322.0> ( -> - Shovel shovel001, vhost: '/', user: 'john.doe'):
client unexpectedly closed TCP connection

=INFO REPORT==== 12-Nov-2020::21:02:06 ===
node rabbit@server1 down: connection_closed

=INFO REPORT==== 12-Nov-2020::21:02:06 ===
node rabbit@server1 up

=ERROR REPORT==== 12-Nov-2020::21:02:06 ===
Partial partition detected:
* We saw DOWN from rabbit@server1
* We can still see rabbit@server3 which can see rabbit@server1
We will therefore intentionally disconnect from rabbit@server3


The rabbitmqctl cluster_status command can be used to view the current nodes in the cluster. In this example, "nodes" shows that there are 3 nodes in the cluster, and "running_nodes" shows that only one of the nodes is "running".

~]# rabbitmqctl cluster_status
Cluster status of node rabbit@server2


The rabbitmqctl node_health_check command can be used.

~]# rabbitmqctl node_health_check
Timeout: 70.0 seconds
Checking health of node rabbit@server2
Health check passed


In this scenario, you can try to restart the node, and then reissue the rabbitmqctl cluster_status command to see if the node is detected as a "running node".


Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee

Add a Comment

We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.

Please enter 02a09 in the box below so that we can be sure you are a human.


Web design by yours truely - me, myself, and I   |   jeremy.canfield@freekb.net   |