RabbitMQ - Resolve "mnesia_event got inconsistent_database running_partitioned_network"

Let's say the main RabbitMQ log file (such as /var/log/rabbitmq/rabbit@server001.log on Linux) contains the following.

=ERROR REPORT==== 14-Dec-2020::18:19:32 ===
Mnesia(rabbit@server001): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@server002}

 

The rabbitmqctl cluster_status command can be used to determine if the servers are missing from the cluster. In this example, server2 is not in running_nodes, meaning that server2 is not in the cluster.

~]# rabbitmqctl cluster_status
Cluster status of node rabbit@server1
[{nodes,[{disc,[rabbit@server1,rabbit@server2,rabbit@server3]}]},
 {running_nodes,[rabbit@server1]},
 {cluster_name,<<"rabbit@server1.example.com">>},
 {partitions,[]},
 {alarms,[{rabbit@server2,[]},{rabbit@server3,[]}]}]

 

The systemctl restart rabbitmq-server command can be used as a quick fix. In this scenario, issuing this command on server1 may temporarily resolve the issue.

There are 4 modes that can be used.

  • ignore (this is the default mode)
  • autoheal
  • pause-minority
  • pause-if-all-down

These modes can be set in the main RabbitMQ configuration file, like this.

[
  {rabbit, [
      {cluster_partition_handling, autoheal}
  ]},

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee

Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.





Please enter fb603 in the box below so that we can be sure you are a human.




Comments

Web design by yours truely - me, myself, and I   |   jeremy.canfield@freekb.net   |