Skip to Content
Last modified by Erik Bakker on 2024/09/03 08:22
From version 17.1
edited by Carlijn Kokkeler
on 2024/08/29 11:05
Change comment: There is no comment for this version
To version 12.1
edited by Erik Bakker
on 2023/01/23 11:49
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.CarlijnKokkeler
1 +XWiki.ebakker
Content
... ... @@ -1,5 +1,5 @@
1 1  {{container}}{{container layoutStyle="columns"}}(((
2 -Understanding retention configuration in Event Streaming is crucial for managing data effectively. This guide will cover how to set and manage retention policies based on both time and size, ensuring you keep control over your topic data. We’ll also explore key considerations, including how retention interacts with topic size, segment calculations, and the impact of data duplication. Use this information to optimize your retention settings across different environments and avoid potential pitfalls.
2 +In this microlearning, we'll look at the impact of changing retention policies for topics across time. A retention policy defines at what thresholds data is deleted from a topic.
3 3  
4 4  Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
5 5  
... ... @@ -49,19 +49,22 @@
49 49  
50 50  When you have to increase your partitions either due to an increase in consumers within a consumer group or to achieve higher throughput in general, there is a migration path you ought to consider. To prevent deletion from kicking in before consumers consume data, we advise first increasing the number of partitions and keeping the retention bytes stable. Consequently, you will temporarily need increased topic storage to facilitate the migration in the safest way possible. Once the data is nicely distributed over the partitions and the consumers have consumed enough data not to lose data before it is consumed, you should divide the current retention bytes setting by the number of partitions to which you have increased that setting. So, for example, if your present retention bytes is 3000 and you increase the number of partitions to three, the new retention bytes setting should be 1000 after the migration is finished.
51 51  
52 -== 4. Key takeaways ==
52 +== 4. Assignment ==
53 53  
54 +Take a moment to review your Event Streaming solution and find the Design Architecture section. Review the additional readings to ensure the management of topics and their retention settings are clearly understood.
55 +
56 +== 5. Key takeaways ==
57 +
54 54  * There are two ways to influence the retention of a topic - hours & bytes
55 55  * Deletion of data in a topic based on retention policy is somewhat complex - be sure to understand the size of a topic before deletion kicks in
56 56  * Be careful in configuring topics so that data is not duplicated unnecessarily and each environment has the right retention policy
57 57  
58 -== 5. Suggested Additional Readings ==
59 -** [[Fundamentals (Navigation)>>doc:Main.eMagiz Academy.Fundamentals.WebHome||target="blank"]]
60 -*** [[Key concepts of Event Streaming (Explanation)>>doc:Main.eMagiz Academy.Fundamentals.fundamental-event-streaming-introduction||target="blank"]]
61 -* [[Intermediate (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.WebHome||target="blank"]]
62 -** [[Key concepts Event Streaming (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Key concepts Event Streaming.WebHome||target="blank"]]
63 -*** [[Understanding retention policy in eMagiz (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Key concepts Event Streaming.intermediate-key-concepts-emagiz-event-streaming-retention-policy||target="blank"]]
64 -** [[Solution Architecture (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Solution Architecture.WebHome||target="blank"]]
65 -*** [[Understanding topic configured size across environments (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Solution Architecture.intermediate-solution-architecture-topic-storage||target="blank"]]
62 +== 6. Suggested Additional Readings ==
66 66  
67 -)))((({{toc/}}))){{/container}}{{/container}}
64 +* [[Understanding retention policy in eMagiz>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Key concepts Event Streaming.intermediate-key-concepts-emagiz-event-streaming-retention-policy||target="blank"]]
65 +* [[Understanding topic configured size across environments>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Solution Architecture.intermediate-solution-architecture-topic-storage||target="blank"]]
66 +* [[Key concepts of Event Streaming>>doc:Main.eMagiz Academy.Fundamentals.fundamental-event-streaming-introduction||target="blank"]]
67 +
68 +== 7. Silent demonstration video ==
69 +
70 +As this is a more theoretical microlearning, we have no video that accompanies this microlearning.)))((({{toc/}}))){{/container}}{{/container}}