RabbitMQ - Resolve "mnesia_event got inconsistent_database running_partitioned_network"

by
Jeremy Canfield |
Updated: December 14 2020
| RabbitMQ articles
Let's say the main RabbitMQ log file (such as /var/log/rabbitmq/rabbit@server001.log on Linux) contains the following.
=ERROR REPORT==== 14-Dec-2020::18:19:32 ===
Mnesia(rabbit@server001): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@server002}
The rabbitmqctl cluster_status command can be used to determine if the servers are missing from the cluster. In this example, server2 is not in running_nodes, meaning that server2 is not in the cluster.
~]# rabbitmqctl cluster_status
Cluster status of node rabbit@server1
[{nodes,[{disc,[rabbit@server1,rabbit@server2,rabbit@server3]}]},
{running_nodes,[rabbit@server1]},
{cluster_name,<<"rabbit@server1.example.com">>},
{partitions,[]},
{alarms,[{rabbit@server2,[]},{rabbit@server3,[]}]}]
The systemctl restart rabbitmq-server command can be used as a quick fix. In this scenario, issuing this command on server1 may temporarily resolve the issue.
There are 4 modes that can be used.
- ignore (this is the default mode)
- autoheal
- pause-minority
- pause-if-all-down
These modes can be set in the main RabbitMQ configuration file, like this.
[
{rabbit, [
{cluster_partition_handling, autoheal}
]},
Did you find this article helpful?
If so, consider buying me a coffee over at