Monitoring consumer lag in Event Streaming

In your Event streaming solution, the role of the consumers is important in several ways. One is to understand whether the consumer is consuming the messages on the topic or needs to catch up in consumption.

Should you have any questions, please get in touch with academy@emagiz.com.

1. Prerequisites

• Advanced knowledge of the eMagiz platform
• Complete relevant Event Streaming microlearnings from Crash course to Intermediate

2. Key concepts

Consumer lag indicates the lag between Kafka producers and consumers. If the data production rate far exceeds the rate at which it is getting consumed, consumer groups will exhibit lag.

It can be understood briefly as the gap between the latest offset and consumer offset. In general, enterprises talk about Kafka, but they refer to the physical Kafka brokers *, a server, either physical or container, that runs Kafka. Brokers are the physical repositories of logs that store and serve Kafka messages. Data storage inside a Kafka broker is done through topics. Topics are divided into partitions, and brokers write data into specific partitions. As the broker writes data - it keeps track of the last offset and records it as the log end offset.

Like Brokers keep track of their write position in each partition, each consumer keeps track of the "read position" in each partition whose data it consumes. It is the only way to track the data it has read.

3. Monitoring lag

Navigate to the Manage phase and select Monitoring. In the left-hand menu, you will find the Event Streaming Statistics section, which is the part we'll need to review the consumer lag. The view below shows an example of the information we display on consumer group level. The shown consumer group is showing a lag of 27 messages which indicates that this consumer still needs to consume the messages. Based on the first offset of the partition, it can be observed that the consumer can still catch up to read all the data from the topic.

4. Key takeaways

• Consumer lag shows how many messages the consumer is falling behind in consumption.
• As part of the daily monitoring, Event Streaming Statistics are available to help you understand whether there is consumer lag.

5. Suggested Additional Readings

If you are interested in this topic and want more information, please read the release notes provided by eMagiz.