Upgrade Mendix Kafka Connector to 1.3.0

Below is a document describing the migration path to update the eMagiz Kafka Connector to the latest version. In this new version, we have improved the producer and consumer configurations to increase the performance and throughput (especially on the producer level). Below we described the steps necessary to migrate correctly and profit from these improvements. For an explanation of how you can configure the eMagiz Kafka connector when you are currently not using it, please check out this microlearning.

Should you have any questions, please get in touch with academy@emagiz.com.

1. Prerequisites

• Advanced knowledge of the eMagiz platform
• A thorough understanding of Mendix and how you currently implement the producer and consumer configuration (including microflows)

2. Key concepts

• This migration path allows you to migrate the eMagiz Kafka connector that is already used within your Mendix project.
• eMagiz Kafka connectors not yet implemented in eMagiz should be configured via the default approach. This approach can be found here.

4. Technical Migration Path

4.1 Import the new eMagiz Kafka Connector

The first step of this migration path is to identify the eMagiz Kafka connector you want to migrate. When this is identified in your Mendix project, you can retrieve the latest version of the eMagiz Kafka Connector via the Mendix Marketplace, import the module, and let Mendix overwrite the configuration. Once this is done, you have two options to finish the migration process.

4.1.1 "Start again from scratch" approach

One approach is to start again from scratch and follow the guidance provided within the server configuration, and this microlearning. Once configured correctly, you must remove the current server configuration to avoid unexpected behavior.

4.1.2 "Update changes manually" approach

With this approach, you keep your current server configuration and microflow(s), but you must manually change the configuration at multiple places to arrive at the correct configuration.

4.1.2.1 Update Producer configuration

Within the producer configuration, which can be found on Server -> Configuration Details -> Producer configurations level, you can select the Producer configuration and press "Edit." As a result, you will see the following pop-up.

Here, you need to change the "batch size" setting to 200000 and "linger ms" to 300.

4.1.2.2 Update Consumer configuration

Within the consumer configuration, which can be found on Server -> Consumers level, you can select the Consumer configuration and press "Edit." Moving to the "Advanced" tab on the following page will show the following configuration details.

Here, you need to change the setting called "fetch min bytes" to 100000.

4.1.2.3 Update "Use cached producer" setting on the "Send to kafka (synchronous)" component

This value needs to be set to true for all microflows that automatically produce messages on a Kafka topic. For the manual test setup, you can leave it to be false. In the default microflows provided with the eMagiz Kafka module, you will see the difference between the "IVK_PublishMessage" microflow and the "IVK_PublishMessage_FromUI."

4.2 Deploy changes

Via the standard process, you can deploy your changes to your environment, and the new configuration settings will take effect in Mendix.

5. Key takeaways

• This migration path allows you to migrate the eMagiz Kafka connector to the latest version.
• You can either "start from scratch again" or "manually implement the correct changes."
• There is a default approach one can follow for Mendix models that currently do not use the eMagiz Kafka connector but want to use it. This approach can be found here.