Changes for page Monitoring partition size
Last modified by Erik Bakker on 2023/09/11 13:59
From version 14.1
edited by Erik Bakker
on 2023/09/11 13:30
on 2023/09/11 13:30
Change comment:
There is no comment for this version
To version 15.1
edited by Erik Bakker
on 2023/09/11 13:59
on 2023/09/11 13:59
Change comment:
There is no comment for this version
Summary
-
Page properties (1 modified, 0 added, 0 removed)
Details
- Page properties
-
- Content
-
... ... @@ -1,5 +1,5 @@ 1 1 {{container}}{{container layoutStyle="columns"}}((( 2 -Topics are a vital part of any Event Streaming solution. Each topic will have an assigned size and understanding the size across time and context is important to ensure the right sizing is chosen for these topics. 2 +Topics are a vital part of any Event Streaming solution. Each topic will have an assigned size, and understanding the size across time and context is important to ensure the right sizing is chosen for these topics. 3 3 4 4 Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]]. 5 5 ... ... @@ -10,25 +10,25 @@ 10 10 11 11 == 2. Key concepts == 12 12 13 -Based on the lines you drew in Capture eMagiz automatically generates a topic. In other words, for each line you dr ew in Capture eMagiz will create an accompanying topic. As a reminder, a topic is a category/feed name to which event records are stored and published. All Kafka records are organized into topics. Producer applications write data to topics and consumer applications read from topics. Records published to the cluster stay in the cluster until a configurable retention period has passed by.13 +Based on the lines you drew in Capture, eMagiz automatically generates a topic. In other words, for each line you draw in Capture, eMagiz will create an accompanying topic. As a reminder, a topic is a category/feed name to which event records are stored and published. All Kafka records are organized into topics. Producer applications write data to topics, and consumer applications read from topics. Records published to the cluster stay in the cluster until a configurable retention period has passed by. 14 14 15 15 == 3. How to monitor partition sizing == 16 16 17 -In the Design phase you can set the required size of apartition. Given the number of partitions eMagiz automatically calculates the topic size. That partition size servesasthe retention size, whicheffectivelymeans that once the topic size reaches that threshold the retention policy will ensure that the oldest messages on the partition are deleted.17 +In the Design phase, you can set the required partition size. Given the number of partitions, eMagiz automatically calculates the topic size. That partition size is the retention size, which means that once the topic size reaches that threshold, the retention policy will ensure that the oldest messages on the partition are deleted. 18 18 19 - It's relevant to understand whether the design size isactuallysufficient to serve the business process for which the topic was createdfor. Incasethere is insufficient partition size, it could mean that some messages are deleted too early andarenot considered for the business process. Futhermore, there are limitsinthe useofpartition size in the eMagiz Cloud. An oversized topic could mean that other topics will not havesufficientstorageavailable for instance.Or some test topics are configured with too much storagewhichnegatively influencesthe overall size.19 +Understanding whether the design size is sufficient to serve the business process for which the topic was created is relevant. If there is insufficient partition size, it could mean that some messages are deleted too early and not considered for the business process. Furthermore, there are limits to partition size in the eMagiz Cloud. An oversized topic could mean that other topics will need more storage. Some test topics are configured with too much storage, negatively influencing the overall size. 20 20 21 -To make the seright decisions and configuration for partition sizing, you can review the actual size of each partition in the Manage phase. Navigate to the Manage phase and select the Monitoring section. In the left21 +To make the right decisions and configuration for partition sizing, you can review the actual size of each partition in the Manage phase. Navigate to the Manage phase and select the Monitoring section. In the left-hand menu, you will see the option Event Streaming Statistics. This page will show an overview of all relevant event streaming information. 22 22 23 23 [[image:Main.Images.Microlearning.WebHome@advanced-monitoring-eventstreaming-monitor-overview.png]] 24 24 25 -The graph on the rightbottom of your screen indicates the partition size and how it evolvesthrough time.25 +The graph on the bottom-right of your screen indicates the partition size and how it evolves. 26 26 27 27 [[image:Main.Images.Microlearning.WebHome@advanced-monitoring-eventstreaming-monitor-partition-size.png]] 28 28 29 29 {{info}} 30 -* Please note that the graph shows the average size (in bytes) of all partitions of a topic per minute.31 -* Thereis afixed value (calledsegment size) in the eMagiz Clusterthatis used as a threshold before the retention policykicks in andstarts to delete data. This value is set to 220 Mb, which means that in case the retention size of the topic is less than 220 Mb the last 220 MB are always available30 +* Please note that the graph shows the average size (in bytes) of all topic partitions per minute. 31 +* A fixed value (segment size) in the eMagiz Cluster is used as a threshold before the retention policy deletes data. This value is set to 220 Mb, which means that in case the retention size of the topic is less than 220 Mb, the last 220 MB are always available 32 32 * In the Design phase, the display of the Assigned topic size takes into account this segment size for calculation purposes 33 33 {{/info}} 34 34 ... ... @@ -35,7 +35,7 @@ 35 35 == 4. Key takeaways == 36 36 37 37 * Topic statistics are available in the Manage - Monitoring section and can be used to see the actual average size of the partitions of a topic. 38 -* Decisions around sizing of partitions can be made using these monitoring data points to determine the optimal configuration of topics across environments 38 +* Decisions around the sizing of partitions can be made using these monitoring data points to determine the optimal configuration of topics across environments 39 39 40 40 == 5. Suggested Additional Readings == 41 41