RabbitMQ - Resolve "missed heartbeats from client"

Let's say the main RabbitMQ log file contains the missed heartbeats from client.

=ERROR REPORT==== 21-Mar-2021::14:22:38 ===
closing AMQP connection <0.14198.805> (10.17.115.16:54343 -> 10.122.15.2:5672 - rabbitConnectionFactory#54ad0027:0):
missed heartbeats from client, timeout: 60s

 

A heartbeat timeout occurs when:

  • An application issues a TCP connection to RabbitMQ
  • The heartbeat timeout is reached (60 seconds by default) before the TCP request is acknowledged

While the TCP request is waiting to be acknowledged, heartbeat frames are sent from the application attempting to connect to RabbitMQ. The heartbeat interval determines how often the heartbeat frames are sent. The heartbeat interval is the heartbeat duration divided by 2. For example, if the heartbeat duration is 60 seconds, then a heartbeat frame would be sent once every 30 seconds.

The rabbitmqctl status command can be used to determine the timeout (in seconds). Notice this output contains both "heartbeat" and "net_ticktime", which actually is a bit confusing, because a net_tick_timeout is different than a heartbeat timeout.

~]# rabbitmqctl status
Cluster heartbeat timeout (net_ticktime): 60

 

The main rabbitmq conf file may have the heartbeat interval defined.

heartbeat = 60

 

Likewise, the heartbeat may be defined in an application. For example, here is how you would set the heartbeat in a Java application. In this example, the heartbeat timeout is set to 5 seconds. The heartbeat defined in an application can be less than the heartbeat defined in RabbitMQ, but not greater.

package com.rabbitmq.demo;

import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.Connection;

public class connect {
  public void amqConnect() {	
    ConnectionFactory cf = new ConnectionFactory();
    cf.setRequestedHeartbeat(5);

 

In this example, the request was sent from the application with IP address 10.17.115.16. The rabbitmqctl list_channels command may be helpful in determing the application that sent the request.

rabbitmqctl list_channels name user

 

Which should return something like this. Notice a unique source port for each channel.

10.17.115.16:54343 -> 10.122.15.2:5672 (1)    john.doe

 

Or, the main RabbitMQ log file may contain the the username being used to connect to RabbitMQ (john.doe in this example).

=INFO REPORT==== 29-Mar-2021::10:01:14 ===
accepting AMQP connection <0.17389.1554> (10.17.115.16:54343 -> 10.122.15.2:5672)

=INFO REPORT==== 29-Mar-2021::10:01:14 ===
Connection <0.17389.1554> (10.17.115.16:54343 -> 10.122.15.2:5672) has a client-provided name: rabbitConnectionFactory#54ad0027:0

=INFO REPORT==== 29-Mar-2021::10:01:14 ===
connection <0.17389.1554> (10.17.115.16:54343 -> 10.122.15.2:5672 - rabbitConnectionFactory#54ad0027:0): user 'john.doe' authenticated and granted access to vhost '/'

 




Did you find this article helpful?

If so, consider buying me a coffee over at Buy Me A Coffee

Add a Comment




We will never share your name or email with anyone. Enter your email if you would like to be notified when we respond to your comment.





Please enter 0292e in the box below so that we can be sure you are a human.




Comments

Web design by yours truely - me, myself, and I   |   jeremy.canfield@freekb.net   |