Changes for page Impact of changing topic retention

Summary

• Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

...	...	@@ -1,5 +1,5 @@
1	1	{{container}}{{container layoutStyle="columns"}}(((
2		-In this microlearning, we'll look at the impact of changing retention policies for topics across time. A retention policy defines at what thresholds data is deleted from a topic.
	2	+In this microlearning, we'll take a look at the impact of changing retention polices for topics across time. A retention policy defines at what thresholds data is deletedf from a topic.
3	3
4	4	Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
5	5
...	...	@@ -10,45 +10,39 @@
10	10
11	11	== 2. Key concepts ==
12	12
13		-Retention configuration in Event Streaming helps to delete data from a topic once that data (messages) reaches one of the retention thresholds. The primary retention parameters that can be influenced in Event Streaming topics are:
	13	+Retention configuration in Event Streaming helps to delete data from a topic once that data (messages) reaches one of the retention thresholds. The main retention parameters that can be influenced in Event Streaming topis are:
14	14
15		-* Retention bytes -> once the topic reaches this size, the oldest messages will be deleted from the topic (FIFO approach)
	15	+* Retention bytes -> once the topic reaches this size the oldest messages will be deleted from the topic (FIFO approach)
16	16	* Retention hours -> once a message on a topic is older than these retention bytes, this message will be deleted
17	17
18		-The cleanup policy is set to <delete>, which means the message is deleted.
	18	+The cleanup policy is set to <delete> which effectively means the message is deleted. The cleanup policy <compact> is not used in eMagiz.
19	19
20		-{{info}}The cleanup policy <compact> is not used in eMagiz.{{/info}}
21		-
22	22	== 3. Considerations for topic retention policy ==
23	23
24	24	=== 3.1 When does the retention kick in? ===
25	25
26		-Retention will only start when the topic size is larger than 220Mb for topics less than 200Mb. The so-called segment size is automatically calculated and is considered for topics larger than 220Mb. However, data may still need to be deleted when the topic size is larger than the retention size configured.
	24	+Retention will only start when the topic size is larger than 220Mb for topics that are less than 200Mb. The socalled segment size is automatically calculated and is considered for topics larger than 220Mb. It may occur that data is not yet deleted at the moment the size of the topic is larger than the retention size configured.
27	27
28	28	=== 3.2 Does the size in Design Architecture consider the segment size? ===
29	29
30		-The total size in eMagiz displayed in the Design phase * Architecture considers the complex calculations around segment sizing, cleanup policies, etc. So your sizing can always be at most that value on our Event Streaming cluster. eMagiz uses this value to determine the required licensing.
	28	+The total size in eMagiz that is displayed in the Design phase * Architecture takes into account the complex calculations around segment sizing, cleanup policies etc. So your sizing can never exceed that value on our Event Streaming cluster. eMagiz uses this value to determine the required licensing.
31	31
32	32	=== 3.3 Actual retention vs. budget retention ===
33	33
34		-Although you set a retention hours setting, the messages can be so large that the retention bytes kick in earlier than the retention hours. This could cause an effect that the oldest messages are younger than your retention hour setting. In specific scenarios, this can be an issue.
	32	+Despite the fact that you set a retention hours setting, it may occur that the amount of messages is so large that the retention bytes kicks in earlier than the retention hours. This could cause an effect that the oldest messages is younger than your retention hour setting. In specific scenario's this can be an issue.
35	35
36	36	=== 3.4 Managing retention across TAP ===
37	37
38		-You can set the retention bytes and hours per Test, Acceptance, and Production. You can play with these configuration values by selecting a smaller value for Test than Production. So that each topic is optimized for the environment at hand. One piece of advice can be to keep Acceptance and Production similar in terms of configuration for retention, whereas the Test can be set to a lower value.
	36	+You are able to set the retention bytes and hours per Test, Acceptance and Production. You can play with these configuration values by setting a smaller value for Test compared to Production. So that each topic is optimized for the environment at hand. One advice can be to keep Acceptance and Production similar in terms of configuration for retention whereas Test can be set to a lower value.
39	39
40		-{{info}}During developing a new business process that involves topics, only the Test can be given sufficient retention values. In this case, Acceptance and Production topics can be disabled so that the total licensed size is not exceeded.{{/info}}
	38	+{{info}}During development of a new business process that involves the use of topics, only test can be given sufficient retention values. In this case, Acceptance and Production topics can be disabled so that the total licensed size is not exceeded.{{/info}}
41	41
42	42	=== 3.5 Duplication of data ===
43	43
44		-* Currently, the standard setting for the number of partitions is one. If you change this value to two or three and keep your retention bytes at the same level, you will have reserved data storage twice as much as before. Consequently, you might run out of the allotted size too fast for comfort. Increasing the number of partitions should result from additional consumer groups with multiple consumers wanting to consume data on a particular topic.
	42	+* At this moment the standard setting for the number of partitions is 1. In case you change this value to 10, then you will have a reserved data storage of 10 times the retention bytes. The grand effect might be that the when adding all topics together including the segment size could increase the total size substantially.
45	45
46		-* When designing solutions in eMagiz using Event Streaming, make sure to make the right decision on storing intermediate results. For example, you could store an intermediate result of a process step in the following topic that then, in turn, feeds the next step in the business process. In this case, using too many topics for the same business process causes an exponential increase in data storage. Try to make a sound decision by combining steps or looking at alternatives.
	44	+* When design solutions in eMagiz using Event Streaming, then make sure to make the right decision on the storage of intermediate results. You could store a intermediate result of a process step in a next topic that then in turn takes the next step in the business process. In this case, it might be that the use of too many topics for the same business process causes an exponential increase of data storage. Try to make a sound decision in this by combining steps or alternative.
47	47
48		-=== 3.6 Increase Partitions ===
49		-
50		-When you have to increase your partitions either due to an increase in consumers within a consumer group or to achieve higher throughput in general, there is a migration path you ought to consider. To prevent deletion from kicking in before consumers consume data, we advise first increasing the number of partitions and keeping the retention bytes stable. Consequently, you will temporarily need increased topic storage to facilitate the migration in the safest way possible. Once the data is nicely distributed over the partitions and the consumers have consumed enough data not to lose data before it is consumed, you should divide the current retention bytes setting by the number of partitions to which you have increased that setting. So, for example, if your present retention bytes is 3000 and you increase the number of partitions to three, the new retention bytes setting should be 1000 after the migration is finished.
51		-
52	52	== 4. Assignment ==
53	53
54	54	Take a moment to review your Event Streaming solution and find the Design Architecture section. Review the additional readings to ensure the management of topics and their retention settings are clearly understood.
...	...	@@ -55,9 +55,9 @@
55	55
56	56	== 5. Key takeaways ==
57	57
58		-* There are two ways to influence the retention of a topic - hours & bytes
	52	+* There are 2 ways to influence the retention of a topic - hours & bytes
59	59	* Deletion of data in a topic based on retention policy is somewhat complex - be sure to understand the size of a topic before deletion kicks in
60		-* Be careful in configuring topics so that data is not duplicated unnecessarily and each environment has the right retention policy
	54	+* Be careful in configuring topics so that data is not duplicated unnecessary and each environment has the right retention policy
61	61
62	62	== 6. Suggested Additional Readings ==
63	63