Topic Storage
In this microlearning, we will dive into the crucial aspect of topic storage in the Event Streaming platform. Building on what you’ve learned about the event streaming pattern, which focuses on topics and retention, well explore how to determine the configured size of topic storage for each environment. Additionally, we’ll discuss steps to take if you exceed your allocated topic storage.
Should you have any questions, please contact academy@emagiz.com.
1. Prerequisites
- Basic knowledge of the eMagiz platform
2. Key concepts
This microlearning centers on topic storage.
With topic storage we mean: The total configured amount of retention on all topics (across environments)
The focal point of this microlearning will be topic storage.
- The key aspects are:
- eMagiz compares the allotted topic storage (contractually allowed) with the configured topic storage
- eMagiz sells topic storage per integration data model (so no separate GBs for Test, Acceptance, and Production)
- eMagiz shows configured topic storage per environment so you can assign yourself how much of the allotted GB you want to use per environment
3. Topic Storage
A key element within using the Event Streaming platform is the topic storage. As you have learned in the Event Streaming Introduction the event streaming pattern is all about topics and retention. As you can imagine the total amount of GB configured in retention across environments and topics defines the total amount of configured topic storage. In this microlearning, we will learn how you can determine the configured size of the topic storage (per environment). Further, we will learn what you should do in case you exceed your allotted amount of topic storage.
Let us first check how we can determine the configured topic storage per enviroment and how you can compare that to the assigned topic storage per environment. To do so we have to navigate towards Design -> Architecture. Within Design Architecture you will a grey block called topic storage.
In this block you will see the configured size on the left (so the total configured retention across topics on this environment) and the assigned (i.e. allotted) size on the right. As long as the number on the right is higher or equals the number on the left you can continue building and deploying your event streaming solution. However the moment the number on the right is lower compared to the number on the left (see below) you have exceeded the assigned size.
You can also see this comparison in the right hand panel on the Design Architecture canvas.
When this happens you could first take a hard look at how you have configured the retention settings per topic. Maybe you will spot large differences between the configured size and the actual retention size on a topic. If so you could downscale the retention settings (in terms of bytes and hours) in order to reduce the number of configured GBs on that environment. Note that you can vary the retention policy of a topic per environment. This allows you to make small Test topics and large(r) Production topics. To figure out whether the configured retention settings match the actual retention on topic level you can use the Manage phase in eMagiz. If you want to learn more on this please check out this Managing your Event Streaming Solution microlearning.
In case you cannot scale down your retention settings you need to contact your partner manager in order to come to an agreement on more allotted topic storage on a contractual basis. Note that just as with cloud approval only an eMagiz cloud administrator can change the assigned configuration. This to prevent misuse of the platform in terms of retention.
4. Key takeaways
- Topic storage is a fundamental component of the Event Streaming platform, directly impacting event retention and system efficiency.
- Understanding how to determine the configured size of topic storage across environments is crucial for managing your resources effectively.
- Knowing the steps to take if you exceed your allocated topic storage ensures you can address potential issues proactively and maintain optimal performance.
5. Suggested Additional Readings
If you are interested in this topic and want more information on it please read the release notes provided by eMagiz and see the following links: