RabbitMQ - Resolve low disk space (mnesia msg_stores)

Typically, on a Linux system, a dedicated partition or volume is used for RabbitMQ. In this example, the /opt/rabbitmq directory has reached 95% used.

~]# df -h
Filesystem                                           Size  Used Avail Use% Mounted on
/dev/mapper/datavg-lv_rabbitmq                        20G   19G  1.2G  95% /opt/rabbitmq

 

The du -hs command can be used. Often, the mnesia directory will be large. mnesia is a database used by RabbitMQ where messages are stored.

~]# du -hs /opt/rabbitmq/*
18G     /opt/rabbitmq/mnesia

 

And often the msg_stores directory will be large. Messages are stored in the msg_stores directory.

~]# du -hs /opt/rabbitmq/mnesia/node1/*
18G     /opt/rabbitmq/mnesia/node1/msg_stores

 

And often the msg_stores_persistent and/or msg_stores_transient directories will be large.

~]# du -hs /opt/rabbitmq/mnesia/node1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/*
13G     /opt/rabbitmq/mnesia/node1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent
4.2G    /opt/rabbitmq/mnesia/node1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient

 

And the msg_stores_persistent and/or msg_stores_transient directories will contain a large collection of .rdq files.

AVOID TROUBLE

A lazy queue will always store messages on disk, in the mnesia database.

Do NOT edit, move, delete or in any way try to tamper with the .rdq files, as these files contain messages.

~]# ls -l /opt/rabbitmq/mnesia/node1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent
-rw-r----- 1 rabbitmq rabbitmq 14581574 Sep 23 10:52 0.rdq
-rw-r----- 1 rabbitmq rabbitmq 16755385 Sep 23 09:14 1016.rdq
-rw-r----- 1 rabbitmq rabbitmq 16749183 Oct  1 11:33 10183.rdq
-rw-r----- 1 rabbitmq rabbitmq 16596144 Oct 26 17:33 10689.rdq
-rw-r----- 1 rabbitmq rabbitmq 15955144 Oct 26 17:33 11214.rdq
-rw-r----- 1 rabbitmq rabbitmq 16720815 Oct  5 22:18 11771.rdq
-rw-r----- 1 rabbitmq rabbitmq 16750081 Oct  6 15:14 12350.rdq
-rw-r----- 1 rabbitmq rabbitmq 16712827 Oct  7 10:43 12630.rdq
-rw-r----- 1 rabbitmq rabbitmq 15322101 Oct  8 07:51 13019.rdq

 

The rabbitmqctl list_queues command can be used to count the number of messages in each queue, as a way to determine which queues contain the most messages. Or the same could be done with the rabbitmqadmin list queues command, or with the RabbitMQ REST API.

In this example, you would want to determine why messages on a.queue and foo.queue and bar.queue are not being consumed.

~]# rabbitmqctl list_queues -p / messages name | sort -n -r | head
619221  a.queue
48882   foo.queue
27561   bar.queue

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee

Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.





Please enter 9e3e9 in the box below so that we can be sure you are a human.




Comments

Web design by yours truely - me, myself, and I   |   jeremy.canfield@freekb.net   |