RabbitMQ - Preventing dead letter message duplication

Let's say you have setup RabbitMQ with max length bytes, so that when a queue reaches max length bytes, messages are moved to a dead letter queue. 

Let's also say the a bunch of messages get published to foo.queue, without being consumed, causing foo.queue to reach it's max length.

 

Let's say 1 more message is published to foo.queue. Assuming RabbitMQ is using the reject-publish-dlx overflow, this will cause the new message to be moved to the dead letter queue. 

Rejected messages get redelivered

Acknowledged (ack) and negatively acknowledged (nack) messages get consumed or dropped

 

In this scenario, when the message is moved to the dead letter queue (deadletter.foo.queue), the redelivered value will be set to false.

This can be seen using the RabbitMQ REST API. In this example, the messages in foo.queue in the default %2f (forward slash) virtual host will be returned.

curl 
--request POST 
--user john.doe:itsasecret 
--data '{"count":10,"ackmode":"ack_requeue_true","encoding":"auto","truncate":50000}'
--url http://server001:15671/api/queues/%2f/foo.queue/get

 

The output should look something like this. Notice that redelivered is false.

[
 {
  "payload_bytes": 6
  "redelivered": false
  "exchange": "foo.exchange"
  "routing_key": "foo.key"
  "message_count": 1
  "properties":{"timestamp":1622715703
  "delivery_mode": 2
  "headers":{"timestamp_in_ms":1622715703577}}
  "payload": "Hello World"
  "payload_encoding": "string"
 }
]

 

Let's say you have the following design, where messages are moved from the dead letter queue back to the original queue.

AVOID TROUBLE

Be aware that a dead letter header exchange bound to a dead letter queue with the x-death[0].count argument will failed to match messages where the count is 1 or 2 or any other integer, as this type of complex matching is not supported by RabbitMQ.

 

In this scenario, when the message gets moved back to the original queue, if the original queue (foo.queue) is still at it's max length, then the message will be moved again to the dead letter queue again, the redelivered value will be set to true, and there is a possibility that the message will be duplicated in the dead letter queue.

[
    {
        "exchange": "deadletter.foo.queue.exchange",
        "message_count": 1,
        "payload": "Hello World",
        "payload_bytes": 6,
        "payload_encoding": "string",
        "properties": {
            "delivery_mode": 1,
            "headers": {
                "timestamp_in_ms": 1636606393468,
                "x-death": [
                    {
                        "count": 1,
                        "exchange": "",
                        "queue": "foo.queue",
                        "reason": "maxlen",
                        "routing-keys": [
                            "foo.queue"
                        ],
                        "time": 1636606393
                    }
                ],
                "x-first-death-exchange": "",
                "x-first-death-queue": "foo.queue",
                "x-first-death-reason": "maxlen"
            },
            "timestamp": 1636606393
        },
        "redelivered": true,
        "routing_key": "*"
    },
    {
        "exchange": "deadletter.foo.queue.exchange",
        "message_count": 0,
        "payload": "Hello World",
        "payload_bytes": 6,
        "payload_encoding": "string",
        "properties": {
            "delivery_mode": 1,
            "headers": {
                "timestamp_in_ms": 1636606393468,
                "x-death": [
                    {
                        "count": 2,
                        "exchange": "",
                        "queue": "foo.queue",
                        "reason": "maxlen",
                        "routing-keys": [
                            "foo.queue"
                        ],
                        "time": 1636606393
                    }
                ],
                "x-first-death-exchange": "",
                "x-first-death-queue": "foo.queue",
                "x-first-death-reason": "maxlen"
            },
            "timestamp": 1636606393
        },
        "redelivered": true,
        "routing_key": "*"
    }
]

 

One way to handle this situation is to have a consumer attached to the dead letter queue, and when a message arrives in the dead letter queue where redelivered is true, the messages could be consumed from the dead letter queue.

 

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee

Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.





Please enter 1db60 in the box below so that we can be sure you are a human.




Comments

Web design by yours truely - me, myself, and I   |   jeremy.canfield@freekb.net   |