Impact of changing topic retention
Understanding retention configuration in Event Streaming is crucial for managing data effectively. This guide will cover how to set and manage retention policies based on both time and size, ensuring you keep control over your topic data. We’ll also explore key considerations, including how retention interacts with topic size, segment calculations, and the impact of data duplication. Use this information to optimize your retention settings across different environments and avoid potential pitfalls.
Should you have any questions, please get in touch with academy@emagiz.com.
1. Prerequisites
- Advanced knowledge of the eMagiz platform
- Complete relevant Event Streaming microlearnings from Crash course to Intermediate
2. Key concepts
Retention configuration in Event Streaming helps to delete data from a topic once that data (messages) reaches one of the retention thresholds. The primary retention parameters that can be influenced in Event Streaming topics are:
- Retention bytes -> once the topic reaches this size, the oldest messages will be deleted from the topic (FIFO approach)
- Retention hours -> once a message on a topic is older than these retention bytes, this message will be deleted
The cleanup policy is set to <delete>, which means the message is deleted.
3. Considerations for topic retention policy
3.1 When does the retention kick in?
Retention will only start when the topic size is larger than 220Mb for topics less than 200Mb. The so-called segment size is automatically calculated and is considered for topics larger than 220Mb. However, data may still need to be deleted when the topic size is larger than the retention size configured.
3.2 Does the size in Design Architecture consider the segment size?
The total size in eMagiz displayed in the Design phase * Architecture considers the complex calculations around segment sizing, cleanup policies, etc. So your sizing can always be at most that value on our Event Streaming cluster. eMagiz uses this value to determine the required licensing.
3.3 Actual retention vs. budget retention
Although you set a retention hours setting, the messages can be so large that the retention bytes kick in earlier than the retention hours. This could cause an effect that the oldest messages are younger than your retention hour setting. In specific scenarios, this can be an issue.
3.4 Managing retention across TAP
You can set the retention bytes and hours per Test, Acceptance, and Production. You can play with these configuration values by selecting a smaller value for Test than Production. So that each topic is optimized for the environment at hand. One piece of advice can be to keep Acceptance and Production similar in terms of configuration for retention, whereas the Test can be set to a lower value.
3.5 Duplication of data
- Currently, the standard setting for the number of partitions is one. If you change this value to two or three and keep your retention bytes at the same level, you will have reserved data storage twice as much as before. Consequently, you might run out of the allotted size too fast for comfort. Increasing the number of partitions should result from additional consumer groups with multiple consumers wanting to consume data on a particular topic.
- When designing solutions in eMagiz using Event Streaming, make sure to make the right decision on storing intermediate results. For example, you could store an intermediate result of a process step in the following topic that then, in turn, feeds the next step in the business process. In this case, using too many topics for the same business process causes an exponential increase in data storage. Try to make a sound decision by combining steps or looking at alternatives.
3.6 Increase Partitions
When you have to increase your partitions either due to an increase in consumers within a consumer group or to achieve higher throughput in general, there is a migration path you ought to consider. To prevent deletion from kicking in before consumers consume data, we advise first increasing the number of partitions and keeping the retention bytes stable. Consequently, you will temporarily need increased topic storage to facilitate the migration in the safest way possible. Once the data is nicely distributed over the partitions and the consumers have consumed enough data not to lose data before it is consumed, you should divide the current retention bytes setting by the number of partitions to which you have increased that setting. So, for example, if your present retention bytes is 3000 and you increase the number of partitions to three, the new retention bytes setting should be 1000 after the migration is finished.
4. Key takeaways
- There are two ways to influence the retention of a topic - hours & bytes
- Deletion of data in a topic based on retention policy is somewhat complex - be sure to understand the size of a topic before deletion kicks in
- Be careful in configuring topics so that data is not duplicated unnecessarily and each environment has the right retention policy