This is second part to the already published Part I which can be found below
Now let's dive a bit deep on the differences between Kafka and RabbitMQ.
- General purpose message broker— Supports request/reply, point to point, and pub-sub communication patterns.
- Smart broker / dumb consumer model— Consistent delivery of messages to consumers.
- Mature platform— Available for Java, .NET, Ruby, Node etc. Also offers many plugins.
- Communication— Can be synchronous or asynchronous.
- Deployment scenarios— Provides distributed deployment scenarios.
- Multi-node cluster to cluster federation— Can form clusters using declaratively or manually.
- High volume publish-subscribe messages and streams platform—Durable, fast and scalable.
- Durable message store— Run in a server cluster, which keeps streams of records in topics.
- Messages— Made up of a value, a key and a timestamp.
- Dumb broker / smart consumer model— Does not try to track which messages are read by consumers and only keeps unread messages. Kafka keeps all messages for a set period of time.
- Requires external services to run— Apache Zookeeper.
RabbitMQ: Push-based approach
RabbitMQ uses a push model where data is sent from Broker to Consumer. This can be used for low latency messaging.
The push model's goal is to distribute messages individually and efficiently, ensuring that work is parallelized equally and messages are handled roughly in the order they arrived in the queue.
Apache Kafka: Pull-based approach
Kafka uses a pull model. Consumers request batches of messages from a specific offset. Pull-based systems have some deficiencies like resources wasting due to polling regularly. Kafka supports a 'long polling' waiting mode until real data comes through to alleviate this drawback.
Because of Kafka's partitions, a pull model makes sense. In a partition with no competing users, Kafka provides message order. This helps users to take advantage of message batching for more efficient message delivery and higher throughput.
- Doesn't support Message Ordering
- Messages are not retained after consumption since it's a queue
- Supports Message priorities
- Provides message ordering because of its partition
- Messages are always there and can be managed using message retention policy
- Doesn't support Message priorities
Compared to message brokers like RabbitMQ, Kafka provides significantly better efficiency. It improves performance by using sequential disc I/O, making it a good option for queue implementation. With limited resources, it can achieve high throughput (millions of messages per second), which is important for big data use cases.
RabbitMQ can also handle a million messages per second, but it does so at the expense of more resources(around 30 nodes). RabbitMQ can be used for many of the same applications as Kafka, but it must be used in combination with other software such as Apache Cassandra.