Creating an event processor with transformation
In this microlearning, we will explore how to create an event processor in eMagiz for handling what we call a transformation case. This event processor will help you efficiently transport, filter, and transform data between two topics. By the end, you will understand how to set up an event processor that can transform data to meet your specific needs
Should you have any questions, please contact academy@emagiz.com.
1. Prerequisites
- Intermediate knowledge of the eMagiz platform
- An Event Streaming License
- Knowledge of the Event Streaming Pattern
- Followed the crash course on Event Streaming
2. Key concepts
This microlearning centers around creating an event processor with transformation.
- By event processor we mean: A flow within eMagiz that consumes data from one topic and transports the data to another topic. In between, you have the option to filter or transform the data (i.e., event).
- By transformation we mean: A method to ensure that the data structure of the input can be changed towards a specific output data structure.
To create an event processor with transformer, we need to have designed an event processor, which we have discussed in the previous microlearning.
3. Creating an event processor with transformation
An Event processor is a flow within eMagiz that retrieves data from one topic (the so-called input topic) and transports the data to another topic (the so-called output topic). During the transport of the data, you can transform the event or filter out events that don't fit your criteria.
In this transformation scenario we will transform the data between the input and the output to ensure the proper data structure for our output.
3.1 Determine Event Processor Design
The first step of creating your event processor is determining whether you have already designed an event processor. Once you have established which event processor that you have designed and needs to be transferred to the Create phase, then you can continue. For this microlearning, we assume that the event processor is already designed.
3.2 Add integrations
The first step is to add the integration to the Create phase of eMagiz. This process of adding flows to Create is the same for each pattern (Messaging, API Gateway, Event Streaming). So simply navigate to the Create phase and select the button "Add integrations" (which is located in the left bottom of the screen).
Here, you select at least one of the topics that needs to go to Create. eMagiz is smart enough to automatically transfer the other as well. You can always select them both, that will work also.
After you have pressed "Save Selection" and everything went according to plan, you will see a pop-up similar to the one that is shown below:
Press the button called "Go to Create" to get back to the Create overview.
3.3 The Create overview
In the overview of the Stream pattern that is presented, you can zoom in on the event processor to see what the input and output topic of the event processor is.
3.4 Event streaming flow
From here we can navigate to the flow by double-clicking on the event processor or via the context menu (accessible via a right mouse click). When you open the flow eMagiz will present you with something as depicted below. A simple flow with an input, an output, two validation components, and a transformation component in between.
This flow is responsible for the following actions:
- Consume data from input topic
- Validate structure of incoming data
- Transform data between input and output
- Validate result of transformation
- Produce data on output topic
Assuming that the transformation needs no changes based on what you have created in Design, the only thing left for you to do is to press "Stop Editing" as eMagiz has already created the complete flow for you.
If you do need to make a change to the transformation you can do so with the guidance from previous microlearnings. When you are finished press "Stop Editing" and create a new version with a good description, ready to be deployed.
You have now successfully created an event processor that is capable of transforming data between the Input Topic and the Output Topic.
4. Key takeaways
- An event processor in eMagiz facilitates the transport of data between an input and output topic.
- By using the transformation functionality, you can adjust the structure of the data output to meet specific your requirements and desired format.
- eMagiz simplifies the creation process with the "Add integrations" functionality, which will transfer your designed event processor to the Create phase effortlessly.
- eMagiz auto-generates the event processor flow. Your main tasks are to verify the transformation, make any necessary adjustments, and finalize by pressing "Save" when satisfied.
5. Suggested Additional Readings
If you are interested in this topic and want more information on it please read the help text provided by eMagiz and read the following links: