Skip to Content
Last modified by Erik Bakker on 2024/09/03 08:22
From version 12.1
edited by Erik Bakker
on 2023/01/23 11:49
Change comment: There is no comment for this version
To version 10.1
edited by Erik Bakker
on 2022/10/03 13:15
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,5 +1,5 @@
1 1  {{container}}{{container layoutStyle="columns"}}(((
2 -In this microlearning, we'll look at the impact of changing retention policies for topics across time. A retention policy defines at what thresholds data is deleted from a topic.
2 +In this microlearning, we'll take a look at the impact of changing retention polices for topics across time. A retention policy defines at what thresholds data is deletedf from a topic.
3 3  
4 4  Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
5 5  
... ... @@ -10,45 +10,39 @@
10 10  
11 11  == 2. Key concepts ==
12 12  
13 -Retention configuration in Event Streaming helps to delete data from a topic once that data (messages) reaches one of the retention thresholds. The primary retention parameters that can be influenced in Event Streaming topics are:
13 +Retention configuration in Event Streaming helps to delete data from a topic once that data (messages) reaches one of the retention thresholds. The main retention parameters that can be influenced in Event Streaming topis are:
14 14  
15 -* Retention bytes -> once the topic reaches this size, the oldest messages will be deleted from the topic (FIFO approach)
15 +* Retention bytes -> once the topic reaches this size the oldest messages will be deleted from the topic (FIFO approach)
16 16  * Retention hours -> once a message on a topic is older than these retention bytes, this message will be deleted
17 17  
18 -The cleanup policy is set to <delete>, which means the message is deleted.
18 +The cleanup policy is set to <delete> which effectively means the message is deleted. The cleanup policy <compact> is not used in eMagiz.
19 19  
20 -{{info}}The cleanup policy <compact> is not used in eMagiz.{{/info}}
21 -
22 22  == 3. Considerations for topic retention policy ==
23 23  
24 24  === 3.1 When does the retention kick in? ===
25 25  
26 -Retention will only start when the topic size is larger than 220Mb for topics less than 200Mb. The so-called segment size is automatically calculated and is considered for topics larger than 220Mb. However, data may still need to be deleted when the topic size is larger than the retention size configured.
24 +Retention will only start when the topic size is larger than 220Mb for topics that are less than 200Mb. The socalled segment size is automatically calculated and is considered for topics larger than 220Mb. It may occur that data is not yet deleted at the moment the size of the topic is larger than the retention size configured.
27 27  
28 28  === 3.2 Does the size in Design Architecture consider the segment size? ===
29 29  
30 -The total size in eMagiz displayed in the Design phase * Architecture considers the complex calculations around segment sizing, cleanup policies, etc. So your sizing can always be at most that value on our Event Streaming cluster. eMagiz uses this value to determine the required licensing.
28 +The total size in eMagiz that is displayed in the Design phase * Architecture takes into account the complex calculations around segment sizing, cleanup policies etc. So your sizing can never exceed that value on our Event Streaming cluster. eMagiz uses this value to determine the required licensing.
31 31  
32 32  === 3.3 Actual retention vs. budget retention ===
33 33  
34 -Although you set a retention hours setting, the messages can be so large that the retention bytes kick in earlier than the retention hours. This could cause an effect that the oldest messages are younger than your retention hour setting. In specific scenarios, this can be an issue.
32 +Despite the fact that you set a retention hours setting, it may occur that the amount of messages is so large that the retention bytes kicks in earlier than the retention hours. This could cause an effect that the oldest messages is younger than your retention hour setting. In specific scenario's this can be an issue.
35 35  
36 36  === 3.4 Managing retention across TAP ===
37 37  
38 -You can set the retention bytes and hours per Test, Acceptance, and Production. You can play with these configuration values by selecting a smaller value for Test than Production. So that each topic is optimized for the environment at hand. One piece of advice can be to keep Acceptance and Production similar in terms of configuration for retention, whereas the Test can be set to a lower value.
36 +You are able to set the retention bytes and hours per Test, Acceptance and Production. You can play with these configuration values by setting a smaller value for Test compared to Production. So that each topic is optimized for the environment at hand. One advice can be to keep Acceptance and Production similar in terms of configuration for retention whereas Test can be set to a lower value.
39 39  
40 -{{info}}During developing a new business process that involves topics, only the Test can be given sufficient retention values. In this case, Acceptance and Production topics can be disabled so that the total licensed size is not exceeded.{{/info}}
38 +{{info}}During development of a new business process that involves the use of topics, only test can be given sufficient retention values. In this case, Acceptance and Production topics can be disabled so that the total licensed size is not exceeded.{{/info}}
41 41  
42 42  === 3.5 Duplication of data ===
43 43  
44 -* Currently, the standard setting for the number of partitions is one. If you change this value to two or three and keep your retention bytes at the same level, you will have reserved data storage twice as much as before. Consequently, you might run out of the allotted size too fast for comfort. Increasing the number of partitions should result from additional consumer groups with multiple consumers wanting to consume data on a particular topic.
42 +* At this moment the standard setting for the number of partitions is 1. In case you change this value to 10 and keep your retention bytes at the same level, you will have a reserved data storage that is 10 times larger as before. As a consequence you might run out of allotted size to fast for your own comfort.
45 45  
46 -* When designing solutions in eMagiz using Event Streaming, make sure to make the right decision on storing intermediate results. For example, you could store an intermediate result of a process step in the following topic that then, in turn, feeds the next step in the business process. In this case, using too many topics for the same business process causes an exponential increase in data storage. Try to make a sound decision by combining steps or looking at alternatives.
44 +* When designing solutions in eMagiz using Event Streaming, make sure to make the right decision on the storage of intermediate results. You could store an intermediate result of a process step in a next topic that then in turn feeds the next step in the business process. In this case, it might be that the use of too many topics for the same business process causes an exponential increase of data storage. Try to make a sound decision in this by combining steps or by looking at alternatives.
47 47  
48 -=== 3.6 Increase Partitions ===
49 -
50 -When you have to increase your partitions either due to an increase in consumers within a consumer group or to achieve higher throughput in general, there is a migration path you ought to consider. To prevent deletion from kicking in before consumers consume data, we advise first increasing the number of partitions and keeping the retention bytes stable. Consequently, you will temporarily need increased topic storage to facilitate the migration in the safest way possible. Once the data is nicely distributed over the partitions and the consumers have consumed enough data not to lose data before it is consumed, you should divide the current retention bytes setting by the number of partitions to which you have increased that setting. So, for example, if your present retention bytes is 3000 and you increase the number of partitions to three, the new retention bytes setting should be 1000 after the migration is finished.
51 -
52 52  == 4. Assignment ==
53 53  
54 54  Take a moment to review your Event Streaming solution and find the Design Architecture section. Review the additional readings to ensure the management of topics and their retention settings are clearly understood.
... ... @@ -55,15 +55,15 @@
55 55  
56 56  == 5. Key takeaways ==
57 57  
58 -* There are two ways to influence the retention of a topic - hours & bytes
52 +* There are 2 ways to influence the retention of a topic - hours & bytes
59 59  * Deletion of data in a topic based on retention policy is somewhat complex - be sure to understand the size of a topic before deletion kicks in
60 -* Be careful in configuring topics so that data is not duplicated unnecessarily and each environment has the right retention policy
54 +* Be careful in configuring topics so that data is not duplicated unnecessary and each environment has the right retention policy
61 61  
62 62  == 6. Suggested Additional Readings ==
63 63  
64 -* [[Understanding retention policy in eMagiz>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Key concepts Event Streaming.intermediate-key-concepts-emagiz-event-streaming-retention-policy||target="blank"]]
65 -* [[Understanding topic configured size across environments>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Solution Architecture.intermediate-solution-architecture-topic-storage||target="blank"]]
66 -* [[Key concepts of Event Streaming>>doc:Main.eMagiz Academy.Fundamentals.fundamental-event-streaming-introduction||target="blank"]]
58 +* [[Understanding retention policy in eMagiz>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Key concepts Event Streaming.intermediate-key-concepts-emagiz-event-streaming-retention-policy.WebHome||target="blank"]]
59 +* [[Understanding topic configured size across environments>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Solution Architecture.intermediate-solution-architecture-topic-storage.WebHome||target="blank"]]
60 +* [[Key concepts of Event Streaming>>doc:Main.eMagiz Academy.Fundamentals.fundamental-event-streaming-introduction.WebHome||target="blank"]]
67 67  
68 68  == 7. Silent demonstration video ==
69 69