Managing your Event Streaming Solution

In this microlearning, we will guide you through effectively managing your event streaming solution on the eMagiz platform. By the end of this microlearning, you will be equipped with the knowledge to monitor and optimize your event streams, ensuring smooth and efficient data flow across your systems.

If you have any questions along the way, feel free to reach out to us at academy@emagiz.com.

1. Prerequisites

• Basic knowledge of the eMagiz platform
• Understanding of the Event Streaming concept
• An active Event Streaming license

2. Key concepts

This microlearning centers around managing your event streaming solution. The eMagiz platform gives you four views on the main dashboard that can help manage your event streaming solution:

• Problematic consumer groups.
  • This way, you can tell whether one or more consumer groups are more than 1000 messages behind.
• Topics.
  • This way, you can tell which topics send metrics.
  • This overview, shows the number of partitions per topic and the number of consumer groups per topic
• Consumer group lag per topic.
  • This way, you can tell whether a specific consumer (group) is lagging and needs to be notified they are to prevent the potential loss of data.
• Message throughput per topic.
  • This way, you can get an indication of the traffic on a specific topic.

3. Managing your Event Streaming Solution - Main Dashboard

Based on the topics you have configured in the previous phases and on the consumers (users with read access) you have configured eMagiz generates four separate overviews for you in the Manage phase of eMagiz. Access this overview by navigating to Manage -> Monitoring -> Event Streaming statistics.

The eMagiz platform gives you four views on the main dashboard that can help manage your event streaming solution:

• Problematic consumer groups.
  • This way, you can tell whether one or more consumer groups are more than 1000 messages behind.
• Topics.
  • This way, you can tell which topics send metrics.
  • This overview, shows the number of partitions per topic and the number of consumer groups per topic
• Consumer group lag per topic.
  • This way, you can tell whether a specific consumer (group) is lagging and needs to be notified they are not processing information fast enough.
• Message throughput per topic.
  • This way, you can get an indication of the traffic on a specific topic.

3.1 Problematic consumer groups

In the table on the top right-hand side of your screen, you see an overview of all problematic consumer groups (i.e. all consumer groups with a consumer lag exceeding 1000 messages). With the help of this table, you can take action and inform the relevant parties that there might be a problem.

3.2 Topics

In the table on the bottom left-hand side, you can see all topics that are deployed on your environment. Per topic you see the number of partitions (which defaults to six), and the number of partitions. With this information you can check whether a topic is correctly deployed on an environment and therefore ready to process messages.

3.3 Consumer group lag - per topic

The third overview segment related to event streaming is at your screen's top left. In this graph, you see the consumer group lag per topic plotted against time. With this information in mind, you can see whether consumer groups can keep up with the demand or not. This can be used as an early indicator to take action before the consumer group becomes problematic.

3.4 Message througput - per topic

The fourth overview is located on the right-bottom hand of your screen. This graph shows the message throughput of each topic. With the help of this graph, you can determine how many messages have been processed by a particular topic within a specific timeframe. This eliminates the need to manually calculate this number to gain insights into message throughput when a specific process behaves unexpectedly.

3. Managing your Event Streaming Solution - Detailed Dashboard

From the main dashboard on event streaming you can zoom in on a specific topic. You can do so by clicking on a topic in the table called "Topics" on the main dashboard. When doing so you see the following overview.
 

Just as on the main dashboard the eMagiz platform gives you four views on the detailed dashboard that can further help manage your event streaming solution:

• Bytes in.
  • This way, you can tell the amount of data flowing in a topic.
• Bytes out.
  • This way, you can tell how much data is consumed by consumers on the selected topic.
• Messaging in.
  • This way, you can tell the number of messaging being placed on the topic.
• Consumer lag per partition.
  • This way, you can tell whether a specific consumer (group) is lagging, on a per-partition basis, and needs to be notified they are not processing information fast enough.

4. Key takeaways

eMagiz provides you with a main and detailed dashboard that can help you manage your event streaming solution:

• Main Dashboard
  • Problematic consumer groups.
    • This way, you can tell whether one or more consumer groups are more than 1000 messages behind.
  • Topics.
    • This way, you can tell which topics send metrics.
    • This overview, shows the number of partitions per topic and the number of consumer groups per topic
  • Consumer group lag per topic.
    • This way, you can tell whether a specific consumer (group) is lagging and needs to be notified they are not processing information fast enough.
  • Message throughput per topic.
    • This way, you can get an indication of the traffic on a specific topic.

• Main Dashboard
  • Bytes in.
    • This way, you can tell the amount of data flowing in a topic.
  • Bytes out.
    • This way, you can tell how much data is consumed by consumers on the selected topic.
  • Messaging in.
    • This way, you can tell the number of messaging being placed on the topic.
  • Consumer lag per partition.
    • This way, you can tell whether a specific consumer (group) is lagging, on a per-partition basis, and needs to be notified they are not processing information fast enough.

5. Suggested Additional Readings

If you are interested in this topic and want more information on it, please read the help text provided by eMagiz when executing these actions and browse through the following links:

• Intermediate Level (Menu)
  • Solution Architecture (Navigation)
    • Topic Storage (Explanation)
• Advanced Level (Menu)
  • Monitoring Event Streaming (Navigation)
• RCA Knowledge Base (Navigation)
  • RCA Knowledge Base Introduction (Explanation)
• Monitoring Event Streaming (Search Results)