Some of the more popular message queue services are:

• Amazon Web Services (AWS) Simple Queue Service (SQS)
• Kafka (this article)
• IBM MQ
• RabbitMQ

Let's say you have a web application that writes to a database. Perhaps there are periodic spikes where there are perhaps millions of write requests causing significant latency. Worse case scenario, this could perhaps causes the web app to load endlessly, fail to load at all, return timeout errors, display wonky errors, yada yada yada. Not good.

message queue to the rescue! The web application could publish messages to the message queue and then the database could consume the messages in queue. This is basically the publish/subscribe architecture (pub/sub), where the web app publishes the message to the queue and the database subscribes to the queue.

Before you can understand what Kafka is, it's helpful to see how data is transmitted without Kafka. In this extremely simplified diagram, the sending system (producer) transmits data to the receiving system (consumer). In this scenario, the data would most probably be in XML format, and be something like an order placed at an online store.

While this is all fine and dandy, this type of design can produce problems. Most notably, if the receiving system is unable to receive the data (eg. the receving system is down or overwhelmed with requests), the sending system may not be able to move onto the next transmission or may continually try to send the data over and over again (which is a waste of the sending systems resources). Kafka resolves these issues.

RabbitMQ sit's between the sending system (producer) and receiving system (consumer), as a sort of broker or proxy. In this way, if the receiving system is unavailable, the sending system does not get trapped in a situation where it cannot move onto it's next transmission or is bogged down trying to send the transmission over and over again. The sending system simply sends the data to Kafka and then moves on (fire and forget). In this way, Kafka makes it so that data tranmission is asynchronous.

Let's install Kafka! Then, start Kafka.