RabbitMQ - Limit queue depth max-length-bytes argument (dead letter)

Before getting into max-length-bytes (and dead letters), let's first look at how a message typically gets transmitted from a producer to a consumer. Typically, the producer puts the message onto an exchange, a routing key is then used to determine what queue the message should be routed to, the messages gets onto the queue, and the consumer gets the message from the queue.

 

This is fine and dandy, but let's say the consumer is unable to get messages off the queue. This could result in a situation where messages accumulate in the queue. Fortunately, RabbitMQ has some options for dealing with this type of situation. One such option is the max-length-bytes argument, to limit the size in bytes of the data (messages) being stored in the queue.

There are numerous ways to set the max-length-bytes argument.

  • Set max-length-bytes as a queue argument
  • Set max-length-bytes as an exchange argument
  • Set max-length-bytes as a policy and apply the policy to one or more queues and/or exchanges
  • Set the x-max-length-bytes argument in an app (such as a Java app)

AVOID TROUBLE

It is important to recognize that once a queue or exchange has been defined, if the queue or exchange does not have the max-length-bytes argument, it is not possible to update the queue or exchange to have the max-length-bytes argument. However, you can create a max-length-bytes policy and then apply the policy to one or more queues/exchanges. For this reason, this article will focus on only the max-length-bytes policy.

 

The rabbitmqctl set_policy command can be used to create a policy for max-length-bytes. In this example, a policy named max_length_bytes_reject_publish is created in virtual host vhost001, the policy is applied to all of the queues (.*) in vhost001, so that each queue can only contain 1048576 bytes (1 MB).

rabbitmqctl set_policy  
--vhost vhost001
--priority 0
--apply-to queues
max_length_reject_publish 
".*" 
'{
  "max-length-bytes": 1048576, 
  "overflow": "reject-publish" 
}'

 

The rabbitmqctl list_policies command can be used to display the policies.

rabbitmqctl list_policies -p vhost001

 

Which should return something like this. Notice overflow is reject-publish.

Listing policies for vhost "vhost001" ...
vhost     name                             pattern   apply-to   definition                                       priority
vhost001  max_length_bytes_reject_publish  .*        queues     {"max-length-bytes":1048576, "overflow":"reject-publish"}   0

 

Let's say queue001 contains 1 MB of message, and you attempt to publish a message to the queue using the web console. With reject-publish, the attempt to publish the message is rejected.

 

On the other hand, let's say the overflow policy is set to drop-head, which is used to drop the oldest message from the queue. drop-head is the default overflow if no overflow is defined. 

To avoid a situation where messages are dropped without having a way to recover the messages, a dead letter exchange and dead letter queue can be setup. The dead letter exchange will be bound to the dead letter queue with the dead letter routing key.  Let's say deadletter.exchange is bound to deadletter.queue with routing key dlrk.

 

AVOID TROUBLE

Each queue/exchange can have only 1 policy. For this reason, it probably makes sense to delete the max_length_bytes_reject_publish policy and then to create a new policy with a better name, such as deadletter.

 

The rabbitmqctl set_policy command again. Notice that this dead letter policy contains both the max-length-bytes and dead letter arguments, and contains the exact name of our dead letter exchange (deadletter.exchange) and dead letter routing key (dlrk).

rabbitmqctl set_policy
--vhost vhost001 
--priority 0
--apply-to queues 
deadletter
".*"
'{
  "max-length-bytes": 1048576, 
  "overflow": "drop-head", 
  "dead-letter-exchange": "deadletter.exchange",
  "dead-letter-routing-key": "dlrk" 
 }'

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee

Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.





Please enter b44fb in the box below so that we can be sure you are a human.




Comments

Web design by yours truely - me, myself, and I   |   jeremy.canfield@freekb.net   |