Skip to Content
Last modified by Erik Bakker on 2024/09/09 12:37
Show last authors
1 {{container}}{{container layoutStyle="columns"}}(((
2 Below is a document describing the migration path to update the eMagiz Kafka Connector to the latest version. In this new version, we have improved the producer and consumer configurations to increase the performance and throughput (especially on the producer level). Below we described the steps necessary to migrate correctly and profit from these improvements. For an explanation of how you can configure the eMagiz Kafka connector when you are currently not using it, please check out this [[microlearning>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Event Streaming Connectors.intermediate-event-streaming-connectors-using-kafka-module-mendix||target="blank"]].
3
4 Should you have any questions, please get in touch with academy@emagiz.com.
5
6 == 1. Prerequisites ==
7
8 * Advanced knowledge of the eMagiz platform
9 * A thorough understanding of Mendix and how you currently implement the producer and consumer configuration (including microflows)
10
11 == 2. Key concepts ==
12
13 * This migration path allows you to migrate the eMagiz Kafka connector that is **already used** within your Mendix project.
14 * eMagiz Kafka connectors not yet implemented in eMagiz should be configured via the default approach. This approach can be found [[here>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Event Streaming Connectors.intermediate-event-streaming-connectors-using-kafka-module-mendix||target="blank"]].
15
16 == 4. Technical Migration Path ==
17
18 === 4.1 Import the new eMagiz Kafka Connector ===
19
20 The first step of this migration path is to identify the eMagiz Kafka connector you want to migrate. When this is identified in your Mendix project, you can retrieve the latest version of the eMagiz Kafka Connector via the Mendix Marketplace, import the module, and let Mendix overwrite the configuration. Once this is done, you have two options to finish the migration process.
21
22 {{info}}Note that due to the fact that we have seperate modules available for Mendix 8 and Mendix 9 the exact latest version you see in the Mendix Marketplace can not match 100% with the version mentioned here. The latest version for Mendix 8 is 1.3.0 and the latest version for Mendix 9 is 2.3.0{{/info}}
23
24 ==== 4.1.1 "Start again from scratch" approach ====
25 One approach is to start again from scratch and follow the guidance provided within the server configuration, and this [[microlearning>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Event Streaming Connectors.intermediate-event-streaming-connectors-using-kafka-module-mendix||target="blank"]]. Once configured correctly, you must remove the current server configuration to avoid unexpected behavior.
26
27 ==== 4.1.2 "Update changes manually" approach ====
28 With this approach, you keep your current server configuration and microflow(s), but you must manually change the configuration at multiple places to arrive at the correct configuration.
29
30 ===== 4.1.2.1 Update Producer configuration ======
31 Within the producer configuration, which can be found on Server -> Configuration Details -> Producer configurations level, you can select the Producer configuration and press "Edit." As a result, you will see the following pop-up.
32
33 [[image:Main.Images.Migrationpath.WebHome@migration-path-migration-path-mendix-kafka-connector-to-130--producer-configuration.png]]
34
35 Here, you need to change the "batch size" setting to 200000 and "linger ms" to 300.
36
37 ===== 4.1.2.2 Update Consumer configuration ======
38 Within the consumer configuration, which can be found on Server -> Consumers level, you can select the Consumer configuration and press "Edit." Moving to the "Advanced" tab on the following page will show the following configuration details.
39
40 [[image:Main.Images.Migrationpath.WebHome@migration-path-migration-path-mendix-kafka-connector-to-130--consumer-configuration.png]]
41
42 Here, you need to change the setting called "fetch min bytes" to 100000.
43
44 ===== 4.1.2.3 Update "Use cached producer" setting on the "Send to kafka (synchronous)" component =====
45
46 This value needs to be set to true for all microflows that automatically produce messages on a Kafka topic. For the manual test setup, you can leave it to be false. In the default microflows provided with the eMagiz Kafka module, you will see the difference between the "IVK_PublishMessage" microflow and the "IVK_PublishMessage_FromUI."
47
48 === 4.2 Deploy changes ===
49
50 Via the standard process, you can deploy your changes to your environment, and the new configuration settings will take effect in Mendix.
51
52 == 5. Key takeaways ==
53
54 * This migration path allows you to migrate the eMagiz Kafka connector to the latest version.
55 * You can either "start from scratch again" or "manually implement the correct changes."
56 * There is a default approach one can follow for Mendix models that currently do not use the eMagiz Kafka connector but want to use it. This approach can be found [[here>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Event Streaming Connectors.intermediate-event-streaming-connectors-using-kafka-module-mendix||target="blank"]].
57
58 == 6. Suggested Additional Readings ==
59
60 * [[Intermediate Level (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.WebHome||target="blank"]]
61 ** [[Event Streaming Connectors (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Event Streaming Connectors.WebHome||target="blank"]]
62 *** [[Using Kafka Module in Mendix (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Event Streaming Connectors.intermediate-event-streaming-connectors-using-kafka-module-mendix||target="blank"]]
63 * [[eMagiz Kafka Connector (Search Result)>>url:https://docs.emagiz.com/bin/view/Main/Search?sort=score&sortOrder=desc&highlight=true&facet=true&r=1&f_space_facet=0%2FMain.&f_type=DOCUMENT&f_locale=en&f_locale=&f_locale=en&text=%22emagiz+kafka+connector%22||target="blank"]]
64 )))((({{toc/}}))){{/container}}{{/container}}