Changes for page Monitoring partition size

Last modified by Erik Bakker on 2023/09/11 13:59
From version 12.1
edited by Erik Bakker
on 2022/10/03 13:19
Change comment: There is no comment for this version
To version 13.1
edited by Erik Bakker
on 2023/09/11 13:29
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -Monitoring Topic size
1 +Monitoring Partition size
Content
... ... @@ -12,33 +12,31 @@
12 12  
13 13  Based on the lines you drew in Capture eMagiz automatically generates a topic. In other words, for each line you drew in Capture eMagiz will create an accompanying topic. As a reminder, a topic is a category/feed name to which event records are stored and published. All Kafka records are organized into topics. Producer applications write data to topics and consumer applications read from topics. Records published to the cluster stay in the cluster until a configurable retention period has passed by.
14 14  
15 -== 3. How to monitor topic sizing ==
15 +== 3. How to monitor partition sizing ==
16 16  
17 -In the Design phase you can set the required size of a topic. That size serves as the retention size, which effectively means that once the topic size reaches that threshold the retention policy will ensure that the oldest messages on the topic are deleted.
17 +In the Design phase you can set the required size of a partition. Given the number of partitions eMagiz automatically calculates the topic size. That partition size serves as the retention size, which effectively means that once the topic size reaches that threshold the retention policy will ensure that the oldest messages on the partition are deleted.
18 18  
19 -It's relevant to understand whether the design size is actually sufficient to serve the business process for which the topic was create for. In case there is insufficient topic size, it could mean that some messages are deleted too early and are not considered for the business process. Futhermore, there are limits in the use of topic size in the eMagiz Cloud. An oversized topic could mean that other topics will not have sufficient storage available for instance. Or some test topics are configured with too much storage which negatively influences the overall size.
19 +It's relevant to understand whether the design size is actually sufficient to serve the business process for which the topic was created for. In case there is insufficient partition size, it could mean that some messages are deleted too early and are not considered for the business process. Futhermore, there are limits in the use of partition size in the eMagiz Cloud. An oversized topic could mean that other topics will not have sufficient storage available for instance. Or some test topics are configured with too much storage which negatively influences the overall size.
20 20  
21 -To make these right decisions and configuration for topic sizing, you can review the actual size of each topic in the Manage phase. Navigate to the Manage phase and select the Monitoring section. In the left hand menu you will the option Topic Statistics. The column partition size displays the total actual size of the topic across the partitions of that topic. In case the topic has more partitions, the value of partition size is the sum of these topics.
21 +To make these right decisions and configuration for partition sizing, you can review the actual size of each partition in the Manage phase. Navigate to the Manage phase and select the Monitoring section. In the left hand menu you will the option Event Streaming Statistics. This page will show an overview of all relevant event streaming information.
22 22  
23 -[[image:Main.Images.Microlearning.WebHome@advanced-monitoring-eventstreaming-monitor-topicsize-1.png]]
23 +[[image:Main.Images.Microlearning.WebHome@advanced-monitoring-eventstreaming-monitor-overview.png]]
24 24  
25 -* Please note that the column retention size is the size per partition - in case you have 5 partitions that value needs to be multiplied by 5 to understand the true total retention size on a topic level
25 +The graph on the right bottom of your screen indicates the partition size and how it evolves through time.
26 +
27 +[[image:Main.Images.Microlearning.WebHome@advanced-monitoring-eventstreaming-monitor-partition-size.png]]
28 +
29 +{{info}}
30 +* Please note that the graph shows the average size (in bytes) of all partitions of a topic per minute.
26 26  * There is a fixed value (called segment size) in the eMagiz Cluster that is used as a threshold before the retention policy kicks in and starts to delete data. This value is set to 220 Mb, which means that in case the retention size of the topic is less than 220 Mb the last 220 MB are always available
27 27  * In the Design phase, the display of the Assigned topic size takes into account this segment size for calculation purposes
33 +{{/info}}
28 28  
29 -== 4. Assignment ==
35 +== 4. Key takeaways ==
30 30  
31 -Take a moment to review your Event Streaming solution and find the Manage * Montoring section to see the topic statistics. Review if you can reproduce the numbers above.
37 +* Topic statistics are available in the Manage - Monitoring section and can be used to see the actual average size of the partitions of a topic.
38 +* Decisions around sizing of partitions can be made using these monitoring data points to determine the optimal configuration of topics across environments
32 32  
33 -== 5. Key takeaways ==
40 +== 5. Suggested Additional Readings ==
34 34  
35 -* Topic statistics are available in the Manage - Monitoring section and can be used to see the actual size of the topic
36 -* Decisions around sizing of topics can be made using these monitoring data points to determine the optimal configuration of topics across environments
37 -
38 -== 6. Suggested Additional Readings ==
39 -
40 -If you are interested in this topic and want more information, please read the release notes provided by eMagiz.
41 -
42 -== 7. Silent demonstration video ==
43 -
44 -As this is a more theoretical microlearning, we have no video that accompanies this microlearning.)))((({{toc/}}))){{/container}}{{/container}}
42 +If you are interested in this topic and want more information, please read the release notes provided by eMagiz.)))((({{toc/}}))){{/container}}{{/container}}