FreeKB - RabbitMQ Resolve "Partial partition detected"
RabbitMQ - Resolve "Partial partition detected"

"Partial partition detected" typically occurs when there is an unexpected TCP communication issue. Thhe following may be displayed in the RabbitMQ admin console.

 

The main RabbitMQ log file (such as /var/log/rabbitmq/rabbit@server002.log on Linux) should contain something like this.

=WARNING REPORT==== 12-Nov-2020::21:02:06 ===
closing AMQP connection <0.23322.0> (10.15.257.11:6359 -> 10.86.115.42:5672 - Shovel shovel001, vhost: '/', user: 'john.doe'):
client unexpectedly closed TCP connection

=INFO REPORT==== 12-Nov-2020::21:02:06 ===
node rabbit@server1 down: connection_closed

=INFO REPORT==== 12-Nov-2020::21:02:06 ===
node rabbit@server1 up

=ERROR REPORT==== 12-Nov-2020::21:02:06 ===
Partial partition detected:
* We saw DOWN from rabbit@server1
* We can still see rabbit@server3 which can see rabbit@server1
We will therefore intentionally disconnect from rabbit@server3

 

The rabbitmqctl cluster_status command can be used to view the current nodes in the cluster. In this example, "nodes" shows that there are 3 nodes in the cluster, and "running_nodes" shows that only one of the nodes is "running".

~]# rabbitmqctl cluster_status
Cluster status of node rabbit@server2
[{nodes,[{disc,[rabbit@server1,rabbit@server2,rabbit@server3]}]},
 {running_nodes,[rabbit@server2]},
 {cluster_name,<<"rabbit@server1.example.com">>},
 {partitions,[]},
 {alarms,[{rabbit@server1,[]},{rabbit@server3,[]}]}]

 

The rabbitmqctl node_health_check command can be used.

~]# rabbitmqctl node_health_check
Timeout: 70.0 seconds
Checking health of node rabbit@server2
Health check passed

 

In this scenario, you can try to restart the node, and then reissue the rabbitmqctl cluster_status command to see if the node is detected as a "running node".

 



Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.




Please enter 1bd25 in the box below so that we can be sure you are a human.




Comments

Web design by yours truely - me, myself, and I   |   jeremy.canfield@freekb.net   |