Skip to Content
Last modified by Erik Bakker on 2024/09/03 08:22
From version 9.1
edited by Erik Bakker
on 2022/10/03 13:11
Change comment: There is no comment for this version
To version 1.1
edited by eMagiz
on 2022/05/05 10:08
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,0 @@
1 -Impact of changing topic retention
Parent
... ... @@ -1,1 +1,0 @@
1 -WebHome
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.ebakker
1 +XWiki.marijn
Default language
... ... @@ -1,1 +1,0 @@
1 -en
Content
... ... @@ -1,8 +1,20 @@
1 -{{container}}{{container layoutStyle="columns"}}(((
1 +{{html wiki="true"}}
2 +<div class="ez-academy">
3 + <div class="ez-academy_body">
4 +
5 +<div class="doc">
6 +
7 +
8 +
9 += Impact of changing topic retention =
10 +
2 2  In this microlearning, we'll take a look at the impact of changing retention polices for topics across time. A retention policy defines at what thresholds data is deletedf from a topic.
3 3  
4 -Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
13 +Should you have any questions, please get in touch with academy@emagiz.com.
5 5  
15 +* Last update: December 2021
16 +* Required reading time: 5 minutes
17 +
6 6  == 1. Prerequisites ==
7 7  
8 8  * Advanced knowledge of the eMagiz platform
... ... @@ -12,11 +12,13 @@
12 12  
13 13  Retention configuration in Event Streaming helps to delete data from a topic once that data (messages) reaches one of the retention thresholds. The main retention parameters that can be influenced in Event Streaming topis are:
14 14  
15 -* Retention bytes -> once the topic reaches this size the oldest messages will be deleted from the topic (FIFO approach)
16 -* Retention hours -> once a message on a topic is older than these retention bytes, this message will be deleted
27 +1. Retention bytes * once the topic reaches this size the oldest messages will be deleted from the topic (FIFO approach)
28 +2. Retention hours * once a message on a topic is older than these retention bytes, this message will be deleted
17 17  
18 18  The cleanup policy is set to <delete> which effectively means the message is deleted. The cleanup policy <compact> is not used in eMagiz.
19 19  
32 +
33 +
20 20  == 3. Considerations for topic retention policy ==
21 21  
22 22  === 3.1 When does the retention kick in? ===
... ... @@ -25,17 +25,17 @@
25 25  
26 26  === 3.2 Does the size in Design Architecture consider the segment size? ===
27 27  
28 -The total size in eMagiz that is displayed in the Design phase * Architecture takes into account the complex calculations around segment sizing, cleanup policies etc. So your sizing can never exceed that value on our Event Streaming cluster. eMagiz uses this value to determine the required licensing.
42 +The total size in eMagiz that is displayed in the Design phase * Architecture takes into account the complex calculations around segment sizing, cleanup policies etc. So your sizing can never exceeed that value on our Event Streaming cluster. eMagiz uses this value to determine the required licensing.
29 29  
30 30  === 3.3 Actual retention vs. budget retention ===
31 31  
32 -Despite the fact that you set a retention hours setting, it may occur that the amount of messages is so large that the retention bytes kicks in earlier than the retention hours. This could cause an effect that the oldest messages is younger than your retention hour setting. In specific scenario's this can be an issue.
46 +Despite the fact that you set a retention hours setting, it may occur that the amount of messages is so large that the retention bytes kicks in earlier than the retention hours. This could cause an effect that the oldest messages is younger than your retention hour setting. In specific scenario's this can be an issues
33 33  
34 34  === 3.4 Managing retention across TAP ===
35 35  
36 -You are able to set the retention bytes and hours per Test, Acceptance and Production. You can play with these configuration values by setting a smaller value for Test compared to Production. So that each topic is optimized for the environment at hand. One advice can be to keep Acceptance and Production similar in terms of configuration for retention whereas Test can be set to a lower value.
50 +You are able to set the retention bytes and hours per Test, Acceptance and Production. You can play with this configuration values by setting a smaller value for Test compared to Production. So that each topic is optimized for the environment at hand. One advice can be to keep Acceptance and Production similar in terms of configuration for retention * test can be set to a lower value.
37 37  
38 -{{info}}During development of a new business process that involves the use of topics, only test can be given sufficient retention values. In this case, Acceptance and Production topics can be disabled so that the total licensed size is not exceeded.{{/info}}
52 +During development of a new business process that involves the use of topics, only test can be given sufficient retention values. In this case, Acceptance and Production topics can be disabled so that the total licensed size is not exceeded.
39 39  
40 40  === 3.5 Duplication of data ===
41 41  
... ... @@ -43,6 +43,8 @@
43 43  
44 44  * When design solutions in eMagiz using Event Streaming, then make sure to make the right decision on the storage of intermediate results. You could store a intermediate result of a process step in a next topic that then in turn takes the next step in the business process. In this case, it might be that the use of too many topics for the same business process causes an exponential increase of data storage. Try to make a sound decision in this by combining steps or alternative.
45 45  
60 +===== Practice =====
61 +
46 46  == 4. Assignment ==
47 47  
48 48  Take a moment to review your Event Streaming solution and find the Design Architecture section. Review the additional readings to ensure the management of topics and their retention settings are clearly understood.
... ... @@ -53,12 +53,21 @@
53 53  * Deletion of data in a topic based on retention policy is somewhat complex - be sure to understand the size of a topic before deletion kicks in
54 54  * Be careful in configuring topics so that data is not duplicated unnecessary and each environment has the right retention policy
55 55  
72 +
73 +
56 56  == 6. Suggested Additional Readings ==
57 57  
58 -* [[Understanding retention policy in eMagiz>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Key concepts Event Streaming.intermediate-key-concepts-emagiz-event-streaming-retention-policy.WebHome||target="blank"]]
59 -* [[Understanding topic configured size across environments>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Solution Architecture.intermediate-solution-architecture-topic-storage.WebHome||target="blank"]]
60 -* [[Key concepts of Event Streaming>>doc:Main.eMagiz Academy.Fundamentals.fundamental-event-streaming-introduction.WebHome||target="blank"]]
76 +1. Understanding retention policy in eMagiz: https://emagiz.github.io/docs/microlearning/intermediate-key-concepts-emagiz-event-streaming-retention-policy
77 +2. Understanding topic configured size across environments: https://emagiz.github.io/docs/microlearning/intermediate-solution-architecture-topic-storage
78 +3. Key concepts of Event Streaming: https://emagiz.github.io/docs/fundamental/fundamental-event-streaming-introduction
61 61  
62 62  == 7. Silent demonstration video ==
63 63  
64 -As this is a more theoretical microlearning, we have no video that accompanies this microlearning.)))((({{toc/}}))){{/container}}{{/container}}
82 +As this is a more theoretical microlearning, we have no video that accompanies this microlearning.
83 +
84 +</div>
85 +</main>
86 +</div>
87 +</div>
88 +
89 +{{/html}}