Topic and Topic Properties
In this microlearning, we will dive into the essentials of topics and their properties within the eMagiz platform. Here, you'll learn about the key aspects of topic configuration, including retention hours, and partitions. This knowledge is crucial to understand how event streaming works. Note that all these settings are generated by default by eMagiz and cannot be altered by users.
If you have any questions along the way, feel free to reach out to us at academy@emagiz.com.
1. Prerequisites
- Basic knowledge of the eMagiz platform
- Understanding of the Event Streaming concept
2. Key concepts
This microlearning centers around topics and the properties of these topics.
- With topic, we mean: A category/feed name to which event records are stored and published.
- With topic properties, we mean: A piece of the whole configuration of a topic.
Knowing what topics are and which properties of these topics you can and should think about before creating them is a crucial part of successfully implementing an event streaming pattern via the eMagiz platform.
Below we will discuss a topic in more depth and zoom in on the topic properties, especially on the retention hours.
After configuring your topic(s) the way you had in mind, you can check your work via the Design Architecture overview.
3. Topic and Topic Properties
Within the eMagiz platform, you can use the Event Streaming pattern to help solve your business case. A crucial part of that solution is the topics and the accompanying properties.
In this section, we will discuss the following:
- What are the topics, and how can I use them.
- Topic properties and their configuration.
3.1 What are topics, and how can I use them
Based on the lines you drew in Capture, eMagiz automatically generates a topic. In other words, for each line you drew in Capture, eMagiz will create an accompanying topic.
As a reminder, a topic is a category/feed name to which event records are stored and published.
As said before, all Kafka records are organized into topics. Producer applications write data to topics, and consumer applications read from topics.
Records published to the cluster stay in the cluster until a configurable retention period has passed by.
Within eMagiz, you can use topics to temporarily store data to make sure that consumers can consume the data at a specific moment in time.
Four characteristics related to topics are:
- Publish/subscribe mechanism
- Asynchronous, realtime
- Dumb broker, smart consumer: Each subscriber can read at their own pace
- Retention
3.2 Topic properties and their configuration
Besides naming the topic, eMagiz also provides you with a set of default settings for your topic. You cannot touch the default settings for replication of these settings. The default values for this are correct. These default settings cover:
- Retention
- Partitioning
- Replication
As a user you can include or exclude topics via the Design Architecture overview in case you do not want a specific topic to exist on a specific environment.
3.2.1 Retention Hours
Retention Hours is the number of hours data can reside on the topic before a FiFo principle of removing the first entry in the log kicks in. However, the moment data is still on the topic beyond this threshold. Therefore, it will automatically start deleting the data.
The default setting eMagiz provides you is 48 hours (two days).
3.2.3 Partitions
In Kafka, the data is stored in topics. The topic will further be divided into multiple partitions. The actual messages or the data will be held in the partition.
By default, your topic will be configured with six partitions. This cannot be altered.
3.2.3.1 Consumer Group Consideration
If you have multiple consumer groups (with numerous consumers), you should consider the effect of this on the partitioning of our broker. A rule of thumb here is that when you have multiple consumers (in one consumer group), the number of partitions should be equal or higher. This could be the case when you run your event processor in a double lane setup or your Mendix application multi-instance, for example. In those cases, one of the consumers in the group will consume data from partition A whereas the second consumer will consume data from partition B. This way, both consumers can only process half the data while the group can still collect the complete data set.
3.4 Exclude topics per environment
In Design Architecture, you can exclude a topic from a specific environment. This way, you can run a particular topic on a specific environment or two of the three environments. This is an easy way to dynamically design your Event Streaming solution to fit your needs per environment.
Navigate Design Architecture and enter the "Start Editing" mode to exclude a topic. Once in this mode, open the context menu on the Topic storage and select the option Edit storage. This will lead you to the following pop-up.
Here, you have a button called "Toggle Exclude." By selecting a topic and pressing the button, you exclude or include a topic (depending on the current state).
4. Key takeaways
- A topic is a category/feed name to which event records are stored and published:
- Retention on this topic is based on a FIFO principle (start at the beginning of the log)
- eMagiz determines the correct settings for each topics per environment
- Topics are automatically generated in eMagiz when you draw the line in Capture
- Check your configuration with the help of Design Architecture to include or exclude topics
5. Suggested Additional Readings
If you are interested in this topic and want more information on it, please read the help text provided by eMagiz when executing these actions and browse through the following links: