Design an Event Processor with Transformation
In this microlearning, we will focus on setting up transformations for event processors. Specifically, we will explore how to convert data from one format to another using eMagiz's transformation functionality. Specifically, we'll focus on linking different message structures between topics in event streaming.
Should you have any questions, please contact academy@emagiz.com.
1. Prerequisites
- Intermediate knowledge of the eMagiz platform
- An Event Streaming License
- Knowledge of the Event Streaming Pattern
- Followed the crash course on Event Streaming
2. Key concepts
This microlearning centers around transforming data within the event processor.
- By event processor we mean: A flow within eMagiz that consumes data from one topic and transports the data to another topic. In between, you have the option to filter or transform the data (i.e., event)
- With a transformation we mean: A method to ensure that the data structure of the input can be changed towards a specific output data structure
The transformation consists of the following:
- Visual description of how certain entities and attributes are transformed from A to B.
- Has a goal to ensure that data flowing from one topic to another is placed on the output topic in the correct data structure.
3. Event Processor Transformation
As with every integration pattern eMagiz gives you the option to use the transformation functionality of eMagiz.
This transformation functionality gives you the option to ensure that when the data structures between input (topic) and output (topic) differ you can transform the data in such a way that what
is placed on the output topic is in the expected data structure.
For Event Streaming the transformation represents how you want to ensure that two message structures can be linked together between topics.
You can access the transformation of an event processor in the Design phase of eMagiz.
To do so you have to navigate to Design and open the context menu on your event processor.
3.1 Defining your message definitions
However, before we can design our transformation we should first define the input and output message definitions.
To do so select the options called Go to input message definition and Go to output message definition.
These options will lead you to an overview that shows the current message definition.
If this is the first time the overview will show the event streaming data model and will indicate that no structure has been defined yet.
With the help of this canvas, you can select which entities and attributes are part of your (input) message definition. Same functionality as with Messaging and API Gateway. Before you can do so, you need to enter "Start Editing" mode.
So for example your (input) message definition could look like this after you have selected entities and attributes for the message definition.
When you are satisfied with your work you can press "Stop Editing" and move to the output message definition.
3.1 Drawing lines for the transformation
After you are done with editing both message definitions you can open the transformation for the event processor via the context menu.
In the overview that will be opened, you will see your message mapping.
With the help of this canvas, you can draw lines to indicate how the input message should be transformed into the output message. Before you can do so, you need to enter "Start Editing" mode.
A finished message mapping (transformation) will look like below:
Congratulations you have successfully created a transformation that will ensure that the input data is outputted in the correct format.
4. Key takeaways
- Event Streaming Transformations represents how to link different message structures between topics using eMagiz's transformation functionality.
- You can create a transformation between two message definitions by drawing lines.
- The approach to transformations is consistent across all integration patterns in eMagiz.
5. Suggested Additional Readings
If you are interested in this topic and want more information on it please read the help text provided by eMagiz and read the following links: