Monitoring consumer lag in Event Streaming
In this microlearning, we'll delve into the concept of consumer lag in Kafka. Consumer lag refers to the delay between data production and consumption, highlighting how far behind a consumer might be in processing messages. We’ll also guide you on how to monitor this lag effectively using eMagiz's Event Streaming Statistics, so you can ensure your consumers are keeping pace with incoming data.
Should you have any questions, please get in touch with academy@emagiz.com.
1. Prerequisites
- Advanced knowledge of the eMagiz platform
- Complete relevant Event Streaming microlearnings from Crash course to Intermediate
2. Key concepts
Consumer lag indicates the lag between Kafka producers and consumers. If the data production rate far exceeds the rate at which it is getting consumed, consumer groups will exhibit lag.
It can be understood briefly as the gap between the latest offset and consumer offset. In general, enterprises talk about Kafka, but they refer to the physical Kafka brokers *, a server, either physical or container, that runs Kafka. Brokers are the physical repositories of logs that store and serve Kafka messages. Data storage inside a Kafka broker is done through topics. Topics are divided into partitions, and brokers write data into specific partitions. As the broker writes data - it keeps track of the last offset and records it as the log end offset.
Like Brokers keep track of their write position in each partition, each consumer keeps track of the "read position" in each partition whose data it consumes. It is the only way to track the data it has read.
3. Monitoring lag
Navigate to the Manage phase and select Monitoring. In the left-hand menu, you will find the Event Streaming Statistics section, which is the part we'll need to review the consumer lag. The view below shows an example of the information we display on consumer group level. The shown consumer group is showing a lag of 27 messages which indicates that this consumer still needs to consume the messages. Based on the first offset of the partition, it can be observed that the consumer can still catch up to read all the data from the topic.
4. Key takeaways
- Consumer lag shows how many messages the consumer is falling behind in consumption.
- As part of the daily monitoring, Event Streaming Statistics are available to help you understand whether there is consumer lag.
5. Suggested Additional Readings
If you are interested in this topic and want more information on it please read the help text provided by eMagiz and read the following links: