Managing your Event Streaming Solution
In this microlearning, we will guide you through effectively managing your event streaming solution on the eMagiz platform. By the end of this microlearning, you will be equipped with the knowledge to monitor and optimize your event streams, ensuring smooth and efficient data flow across your systems.
If you have any questions along the way, feel free to reach out to us at academy@emagiz.com.
1. Prerequisites
- Basic knowledge of the eMagiz platform
- Understanding of the Event Streaming concept
- An active Event Streaming license
2. Key concepts
This microlearning centers around managing your event streaming solution. The eMagiz platform gives you six views that can help manage your event streaming solution:
- Per topic overview of the bytes in and bytes out.
- This way, you can tell something about the load on a specific topic.
- Per partition overview of the messages in the partition size and the offset count.
- This way, you can tell whether data is evenly distributed, whether the partition size is edging the limit, and how many messages are retained on the topic.
- Per topic (or partition), per consumer group overview, how many messages that consumer group is behind.
- This way, you can tell whether a specific consumer (group) is lagging and needs to be notified they are to prevent the potential loss of data.
3. Managing your Event Streaming Solution
Based on the topics you have configured in the previous phases and on the consumers (external users with read access) you have configured eMagiz generates six separate overviews for you in the Manage phase of eMagiz. Access this overview by navigating to Manage -> Monitoring -> Event Streaming statistics.
This overview consists of six perspectives on your configuration and actual use of the topic that can help you manage your event streaming solution:
- Per topic overview of the bytes in and bytes out.
- This way, you can tell something about the load on a specific topic.
- Per partition overview of the messages in the partition size and the offset count.
- This way, you can tell whether data is evenly distributed, whether the partition size is edging the limit, and how many messages are retained on the topic.
- Per topic (or partition), per consumer group overview, how many messages that consumer group is behind.
- This way, you can tell whether a specific consumer (group) is lagging and needs to be notified they are to prevent the potential loss of data.
3.1 Per topic overview (bytes in/out)
In the two graphs on the top right side of your screen, you see an overview of all bytes in/out on the topic level related to your event streaming solution on your specific environment. With the help of these graphs, you can see how much traffic flows through a particular topic and determine the load on a topic.
3.2 Per partition overview of the messages in, the partition size and the offset count
In the graphs on the bottom right-hand side and the table on the bottom right-hand side, you can see the number of messages produced on specific partitions, the size of partitions, and in the table, the offset count on partition level. This information tells you whether your producers are nicely distributing the messages round-robin across all partitions and whether a partition limit (set in Design) is about to be reached. On top of that, it gives you a general sense of how many messages are retained on each partition.
This information is beneficial for comparing the Actual vs. Budget retention size.
In other words, with the help of these information points, you can calculate the actual retention bytes of a topic.
Sum up the retention bytes per partition and the actual retention size.
That information lets you see how much it differs from the configured retention size.
When there is a big gap between what is configured and what is used, you should adapt your configuration based on what is used.
You must do so to avoid running out of Topic Storage for other topics, so you cannot extend your event streaming solution further.
3.3 Per topic (or partition), per consumer group overview, how many messages that consumer group is behind
The third overview segment related to event streaming is at your screen's top left. In this table, you see an overview of consumer groups and, per topic (or partition), the lag of the consumer group compared to what is already produced on the topic. With this information in mind, you can see whether the lag is becoming too large, causing the consumer group to (potentially) lose data.
In Event Streaming terms, we call this consumer lag. Measuring consumer lag gives you insight into whether consumers are quick or slow to consume data.
Based on this information, you could reduce or increase the number of hours a record stays on your topic.
For example, if you notice that all consumers consume data within the hour, there is no reason to hold that record for seven days.
Configuring this as close as possible (with safety margins obviously) to the reality (measured over an extended period) can help control data on the topic.
4. Key takeaways
eMagiz provides you with three separate overviews that can help you manage your event streaming solution:
- Per topic overview of the bytes in and bytes out.
- This way, you can tell something about the load on a specific topic.
- Per partition overview of the messages in the partition size and the offset count.
- This way, you can tell whether data is evenly distributed, whether the partition size is edging the limit, and how many messages are retained on the topic.
- Per topic (or partition), per consumer group overview, how many messages that consumer group is behind.
- This way, you can tell whether a specific consumer (group) is lagging and needs to be notified they are to prevent the potential loss of data.
Based on these overviews, you can compare what you have configured and what the reality is regarding retention size, retention hours, and consumer group configuration.
5. Suggested Additional Readings
If you are interested in this topic and want more information on it, please read the help text provided by eMagiz when executing these actions and browse through the following links: