Skip to Content
Last modified by Erik Bakker on 2024/09/03 08:22
From version 7.1
edited by Erik Bakker
on 2022/06/13 10:21
Change comment: There is no comment for this version
To version 2.2
edited by Erik Bakker
on 2022/06/13 10:15
Change comment: Update document after refactoring.

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -Impact of changing topic retention
1 +advanced-keyconcepts-eventstreaming-impact-retention
Content
... ... @@ -1,8 +1,13 @@
1 1  {{container}}{{container layoutStyle="columns"}}(((
2 += Impact of changing topic retention =
3 +
2 2  In this microlearning, we'll take a look at the impact of changing retention polices for topics across time. A retention policy defines at what thresholds data is deletedf from a topic.
3 3  
4 4  Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
5 5  
8 +* Last update: December 2021
9 +* Required reading time: 5 minutes
10 +
6 6  == 1. Prerequisites ==
7 7  
8 8  * Advanced knowledge of the eMagiz platform
... ... @@ -12,11 +12,13 @@
12 12  
13 13  Retention configuration in Event Streaming helps to delete data from a topic once that data (messages) reaches one of the retention thresholds. The main retention parameters that can be influenced in Event Streaming topis are:
14 14  
15 -* Retention bytes -> once the topic reaches this size the oldest messages will be deleted from the topic (FIFO approach)
16 -* Retention hours -> once a message on a topic is older than these retention bytes, this message will be deleted
20 +1. Retention bytes * once the topic reaches this size the oldest messages will be deleted from the topic (FIFO approach)
21 +2. Retention hours * once a message on a topic is older than these retention bytes, this message will be deleted
17 17  
18 18  The cleanup policy is set to <delete> which effectively means the message is deleted. The cleanup policy <compact> is not used in eMagiz.
19 19  
25 +
26 +
20 20  == 3. Considerations for topic retention policy ==
21 21  
22 22  === 3.1 When does the retention kick in? ===
... ... @@ -43,6 +43,8 @@
43 43  
44 44  * When design solutions in eMagiz using Event Streaming, then make sure to make the right decision on the storage of intermediate results. You could store a intermediate result of a process step in a next topic that then in turn takes the next step in the business process. In this case, it might be that the use of too many topics for the same business process causes an exponential increase of data storage. Try to make a sound decision in this by combining steps or alternative.
45 45  
53 +
54 +
46 46  == 4. Assignment ==
47 47  
48 48  Take a moment to review your Event Streaming solution and find the Design Architecture section. Review the additional readings to ensure the management of topics and their retention settings are clearly understood.
... ... @@ -53,11 +53,13 @@
53 53  * Deletion of data in a topic based on retention policy is somewhat complex - be sure to understand the size of a topic before deletion kicks in
54 54  * Be careful in configuring topics so that data is not duplicated unnecessary and each environment has the right retention policy
55 55  
65 +
66 +
56 56  == 6. Suggested Additional Readings ==
57 57  
58 -* [[Understanding retention policy in eMagiz>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Key concepts Event Streaming.intermediate-key-concepts-emagiz-event-streaming-retention-policy.WebHome||target="blank"]]
59 -* [[Understanding topic configured size across environments>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Solution Architecture.intermediate-solution-architecture-topic-storage.WebHome||target="blank"]]
60 -* [[Key concepts of Event Streaming>>doc:Main.eMagiz Academy.Fundamentals.fundamental-event-streaming-introduction.WebHome||target="blank"]]
69 +1. Understanding retention policy in eMagiz: https://emagiz.github.io/docs/microlearning/intermediate-key-concepts-emagiz-event-streaming-retention-policy
70 +2. Understanding topic configured size across environments: https://emagiz.github.io/docs/microlearning/intermediate-solution-architecture-topic-storage
71 +3. Key concepts of Event Streaming: https://emagiz.github.io/docs/fundamental/fundamental-event-streaming-introduction
61 61  
62 62  == 7. Silent demonstration video ==
63 63