Using Mendix as consumer

(((

Please note that the following only applies to our latest version of the eMagiz Connector (1.1.0). If you wish to migrate to this version, please review the migration path available in the eMagiz platform.

6

7

8

In this microlearning, we will focus on how you can utilize the eMagiz Connector, available via the portal, to consume data from topics managed within the eMagiz Kafka Cluster. By the end, you'll have a clear understanding of setting up connections, configuring consumers, and utilizing the eMagiz Connector to streamline data transportation between systems.

9

10

Should you have any questions, please get in touch with us at [[academy@emagiz.com>>mailto:academy@emagiz.com]].

11

12

== 1. Prerequisites ==

13

14

* Basic knowledge of the eMagiz platform

15

* Basic knowledge of the Mendix platform

16

* Mendix project in which you can test this functionality

17

* A Kafka cluster to which you can connect.

18

* Access to the Deploy phase of your eMagiz project

19

* A connection between Mendix and eMagiz Designed, Created, and in the **active release**

20

* Using the "eMagiz Connector" in the Mendix Marketplace.

21

* Initial [[installation>>doc:Main.eMagiz Academy.Microlearnings.Novice.Mendix Connectivity.novice-mendix-connectivity-install-emagiz-mendix-connector.WebHome||target="blank"]] done.

22

* Initial [[configuration>>doc:Main.eMagiz Academy.Microlearnings.Novice.Mendix Connectivity.novice-mendix-connectivity-configure-emagiz-mendix-connector.WebHome||target="blank"]] done.

23

24

== 2. Key concepts ==

25

26

This microlearning focuses on using the eMagiz Connector in Mendix.

27

* By using, in this context, we mean: Being able to consume data from topics that are managed within the eMagiz Kafka Cluster.

28

* By knowing how you can easily set up Mendix to consume data from topics, you have the option to better transport large volumes of data between several systems (i.e., two Mendix applications).

29

30

Some key concepts before you dive further into this microlearning:

31

* Consuming data from a topic means that the external system, in this case, Mendix, reads data from a pre-defined topic where the data is stored temporarily.

32

33

== 3. Using the eMagiz Connector Module ==

34

35

With the help of the Mendix module (called eMagiz Connector) created by the eMagiz team, you can easily connect between Mendix and eMagiz for data integration. This microlearning will focus on consuming data via a connection to the eMagiz Kafka Cluster. The previos microlearning focused on producing data.

36

37

To use the eMagiz Connector in Mendix, you need to be able to do at least the following:

38

39

* Set up a connection to the external Kafka Cluster (i.e., the eMagiz Kafka Cluster) from Mendix

40

* Configure a consumer that can read (listen) data from a topic

41

42

Once you have configured these steps, consider how you want to transfer data from and to your data model. That part is excluded from this microlearning, as it focuses solely on how to build microflows in Mendix.

43

44

In the remainder of this microlearning, we will discuss the options through which you can functionally receive messages utilizing the eMagiz Kafka cluster.

45

46

=== 3.1 Consuming Messages - "String Variant" ===

47

48

To send data from Mendix, you must define several configuration options in the Mendix project. In this example, we will examine a simple configuration using a string variable to represent the payload. Later, we examine the variant using a Mendix object as the starting point for sending data.

49

50

In eMagiz, you will see a flow in Create that you cannot edit, as the autogenerated functionality provided by eMagiz is all that is needed. Once done, ensure the flow is transported to Deploy and is ready to be used in a Mendix application by adding it to a release.

51

52

{{info}}The entry version must be altered when you change your system messages in Design.{{/info}}

53

54

In parallel, the Mendix application must be developed to handle the creation of a message and the ability to send it to the eMagiz Kafka cluster, whether earlier or later.

55

56

{{warning}}Whenever you are asked to define the "Protocol," select "KAFKA" if you want to connect to our eMagiz Kafka cluster.{{/warning}}

57

58

==== 3.1.1 Startup action ====

59

60

In Mendix, you must add the "StartStringConsumer" Java Action to the after startup flow. The most important configuration items are as follows.

61

62

[[image:Main.Images.Microlearning.WebHome@intermediate-event-streaming-connectors-using-kafka-module-mendix--start-consumer.png]]

63

64

* The destination (i.e., the topic from which you want to consume the **request** messages). This can be supplied by the eMagiz developer or found under Deploy -> Mendix Connectors should you have access to the eMagiz model in question.

65

* Validate message. This determines whether the incoming message is validated **before** send to the microflow that handles the incoming message.

66

* The microflow that handles the incoming message.

67

* Auto offset reset option. This defaults to the advised setting (i.e. Latest) and is important to consider. This setting determines where the consumer group will start once it is connected for the first time to the broker.

68

69

=== 3.2 Consuming Messages - "Object Variant" ===

70

71

To send data from Mendix, you must define several configuration options in the Mendix project. In this example, we will examine the complex configuration using a Mendix object in conjunction with an export mapping that represents the payload.

72

73

In eMagiz, you will see a flow in Create that you cannot edit, as the autogenerated functionality provided by eMagiz is all that is needed. Once done, ensure the flow is transported to Deploy and is ready to be used in a Mendix application by adding it to a release.

74

75

{{info}}The entry version must be altered when you change your system messages in Design.{{/info}}

76

77

In parallel, the Mendix application must be developed to handle the creation of a message and the ability to send it to the eMagiz Kafka cluster, whether earlier or later.

78

79

{{warning}}Whenever you are asked to define the "Protocol," select "KAFKA" if you want to connect to our eMagiz Kafka cluster.{{/warning}}

80

81

==== 3.1.1 Startup action ====

82

83

In Mendix, you must add the "StartMessageConsumer" Java Action to the after startup flow. The most important configuration items are as follows.

84

85

[[image:Main.Images.Microlearning.WebHome@intermediate-event-streaming-connectors-using-kafka-module-mendix--start-consumer-message.png]]

86

87

* The destination (i.e., the topic from which you want to consume the messages). This can be supplied by the eMagiz developer, found under Deploy -> Mendix Connectors, should you have access to the eMagiz model in question, or viewed in the eMagiz Catalog.

88

* The import mapping (the mapping of the incoming JSON or XML payload to your data entities). The definition of the JSON/XML can be determined by the eMagiz developer (and then shared with the Mendix developer) or determined by the Mendix developer (and then shared with the eMagiz developer).

89

* Validate message. This determines whether the incoming message is validated **before** send to the microflow that handles the incoming message.

90

* The microflow that handles the incoming message.

91

* Auto offset reset option. This defaults to the advised setting (i.e. Latest) and is important to consider. This setting determines where the consumer group will start once it is connected for the first time to the broker.

92

93

== 4. Key takeaways ==

94

95

To listen to a topic in Mendix, you need the following:

96

97

* The destination (i.e., the topic from which you want to retrieve the information)

98

* Execute microflow (the microflow that will process the data that is received in Mendix)

99

* Optionally, you can define an import mapping and subsequent options.

100

101

{{warning}}To make a connection in allowing data to travel between A and B, you need to [[install>>doc:Main.eMagiz Academy.Microlearnings.Novice.Mendix Connectivity.novice-mendix-connectivity-install-emagiz-mendix-connector.WebHome||target="blank"]] and do a basic [[configuration>>doc:Main.eMagiz Academy.Microlearnings.Novice.Mendix Connectivity.novice-mendix-connectivity-configure-emagiz-mendix-connector.WebHome||target="blank"]] before you can set up this specific communication.{{/warning}}

102

103

== 5. Suggested Additional Readings ==

104

105

If you are interested in this topic and want more information on it, please see the following links:

106

107

* [[Fundamentals (Navigation)>>doc:Main.eMagiz Academy.Fundamentals.WebHome||target="blank"]]

108

** [[eMagiz Event Streaming (Explanation)>>doc:Main.eMagiz Academy.Fundamentals.fundamental-event-streaming-introduction||target="blank"]]

109

* [[eMagiz Support (Menu)>>doc:Main.eMagiz Support.WebHome||target="blank"]]

110

** [[Migration Paths (Navigation)>>doc:Main.eMagiz Support.Migration Paths.WebHome||target="blank"]]

111

*** [[Upgrade Mendix Kafka Connector to 1.3.0 (Explanation)>>doc:Main.eMagiz Support.Migration Paths.migration-path-mendix-kafka-connector-to-130.WebHome||target="blank"]]

112

* [[Crash Courses (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.WebHome||target="blank"]]

113

** [[Crash Course Event Streaming (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Event Streaming.WebHome||target="blank"]]

114

*** [[Event Catalog (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Event Streaming.crashcourse-eventstreaming-catalog||target="blank"]]

115

* [[EMagiz Connector (Search Results)>>url:https://docs.emagiz.com/bin/view/Main/Search?sort=score&sortOrder=desc&highlight=true&facet=true&r=1&f_space_facet=0%2FMain.&l_space_facet=10&f_type=DOCUMENT&f_locale=en&f_locale=&f_locale=en&text=%22emagiz+connector%22||target="blank"]]

116

* [[Kafka Explained (External)>>https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html#::text=Apache%20Kafka%20is%20a%20publish||target="blank"]]

117

* [[Topic Configuration (External)>>https://kafka.apache.org/documentation/#topicconfigs||target="blank"]]

118

* [[Kafka sizing (External)>>https://medium.com/@tsureshkumar/sizing-kafka-capacity-needed-for-your-application-fdb6f24f67cd||target="blank"]]

119

* [[Choose Partition Numbers (External)>>https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/||target="blank"]]

120

* [[Kafka Apache Introduction (External)>>https://kafka.apache.org/intro||target="blank"]]

)))

(((

)))

author	version	line-number	content
		1	{{container}}
		2	{{container layoutStyle="columns"}}
		3	(((
		4	{{warning}}
		5	Please note that the following only applies to our latest version of the eMagiz Connector (1.1.0). If you wish to migrate to this version, please review the migration path available in the eMagiz platform.
		6	{{/warning}}
		7
		8	In this microlearning, we will focus on how you can utilize the eMagiz Connector, available via the portal, to consume data from topics managed within the eMagiz Kafka Cluster. By the end, you'll have a clear understanding of setting up connections, configuring consumers, and utilizing the eMagiz Connector to streamline data transportation between systems.
		9
		10	Should you have any questions, please get in touch with us at [[academy@emagiz.com>>mailto:academy@emagiz.com]].
		11
		12	== 1. Prerequisites ==
		13
		14	* Basic knowledge of the eMagiz platform
		15	* Basic knowledge of the Mendix platform
		16	* Mendix project in which you can test this functionality
		17	* A Kafka cluster to which you can connect.
		18	* Access to the Deploy phase of your eMagiz project
		19	* A connection between Mendix and eMagiz Designed, Created, and in the active release
		20	* Using the "eMagiz Connector" in the Mendix Marketplace.
		21	* Initial [[installation>>doc:Main.eMagiz Academy.Microlearnings.Novice.Mendix Connectivity.novice-mendix-connectivity-install-emagiz-mendix-connector.WebHome\|\|target="blank"]] done.
		22	* Initial [[configuration>>doc:Main.eMagiz Academy.Microlearnings.Novice.Mendix Connectivity.novice-mendix-connectivity-configure-emagiz-mendix-connector.WebHome\|\|target="blank"]] done.
		23
		24	== 2. Key concepts ==
		25
		26	This microlearning focuses on using the eMagiz Connector in Mendix.
		27	* By using, in this context, we mean: Being able to consume data from topics that are managed within the eMagiz Kafka Cluster.
		28	* By knowing how you can easily set up Mendix to consume data from topics, you have the option to better transport large volumes of data between several systems (i.e., two Mendix applications).
		29
		30	Some key concepts before you dive further into this microlearning:
		31	* Consuming data from a topic means that the external system, in this case, Mendix, reads data from a pre-defined topic where the data is stored temporarily.
		32
		33	== 3. Using the eMagiz Connector Module ==
		34
		35	With the help of the Mendix module (called eMagiz Connector) created by the eMagiz team, you can easily connect between Mendix and eMagiz for data integration. This microlearning will focus on consuming data via a connection to the eMagiz Kafka Cluster. The previos microlearning focused on producing data.
		36
		37	To use the eMagiz Connector in Mendix, you need to be able to do at least the following:
		38
		39	* Set up a connection to the external Kafka Cluster (i.e., the eMagiz Kafka Cluster) from Mendix
		40	* Configure a consumer that can read (listen) data from a topic
		41
		42	Once you have configured these steps, consider how you want to transfer data from and to your data model. That part is excluded from this microlearning, as it focuses solely on how to build microflows in Mendix.
		43
		44	In the remainder of this microlearning, we will discuss the options through which you can functionally receive messages utilizing the eMagiz Kafka cluster.
		45
		46	=== 3.1 Consuming Messages - "String Variant" ===
		47
		48	To send data from Mendix, you must define several configuration options in the Mendix project. In this example, we will examine a simple configuration using a string variable to represent the payload. Later, we examine the variant using a Mendix object as the starting point for sending data.
		49
		50	In eMagiz, you will see a flow in Create that you cannot edit, as the autogenerated functionality provided by eMagiz is all that is needed. Once done, ensure the flow is transported to Deploy and is ready to be used in a Mendix application by adding it to a release.
		51
		52	{{info}}The entry version must be altered when you change your system messages in Design.{{/info}}
		53
		54	In parallel, the Mendix application must be developed to handle the creation of a message and the ability to send it to the eMagiz Kafka cluster, whether earlier or later.
		55
		56	{{warning}}Whenever you are asked to define the "Protocol," select "KAFKA" if you want to connect to our eMagiz Kafka cluster.{{/warning}}
		57
		58	==== 3.1.1 Startup action ====
		59
		60	In Mendix, you must add the "StartStringConsumer" Java Action to the after startup flow. The most important configuration items are as follows.
		61
		62	[[image:Main.Images.Microlearning.WebHome@intermediate-event-streaming-connectors-using-kafka-module-mendix--start-consumer.png]]
		63
		64	* The destination (i.e., the topic from which you want to consume the request messages). This can be supplied by the eMagiz developer or found under Deploy -> Mendix Connectors should you have access to the eMagiz model in question.
		65	* Validate message. This determines whether the incoming message is validated before send to the microflow that handles the incoming message.
		66	* The microflow that handles the incoming message.
		67	* Auto offset reset option. This defaults to the advised setting (i.e. Latest) and is important to consider. This setting determines where the consumer group will start once it is connected for the first time to the broker.
		68
		69	=== 3.2 Consuming Messages - "Object Variant" ===
		70
		71	To send data from Mendix, you must define several configuration options in the Mendix project. In this example, we will examine the complex configuration using a Mendix object in conjunction with an export mapping that represents the payload.
		72
		73	In eMagiz, you will see a flow in Create that you cannot edit, as the autogenerated functionality provided by eMagiz is all that is needed. Once done, ensure the flow is transported to Deploy and is ready to be used in a Mendix application by adding it to a release.
		74
		75	{{info}}The entry version must be altered when you change your system messages in Design.{{/info}}
		76
		77	In parallel, the Mendix application must be developed to handle the creation of a message and the ability to send it to the eMagiz Kafka cluster, whether earlier or later.
		78
		79	{{warning}}Whenever you are asked to define the "Protocol," select "KAFKA" if you want to connect to our eMagiz Kafka cluster.{{/warning}}
		80
		81	==== 3.1.1 Startup action ====
		82
		83	In Mendix, you must add the "StartMessageConsumer" Java Action to the after startup flow. The most important configuration items are as follows.
		84
		85	[[image:Main.Images.Microlearning.WebHome@intermediate-event-streaming-connectors-using-kafka-module-mendix--start-consumer-message.png]]
		86
		87	* The destination (i.e., the topic from which you want to consume the messages). This can be supplied by the eMagiz developer, found under Deploy -> Mendix Connectors, should you have access to the eMagiz model in question, or viewed in the eMagiz Catalog.
		88	* The import mapping (the mapping of the incoming JSON or XML payload to your data entities). The definition of the JSON/XML can be determined by the eMagiz developer (and then shared with the Mendix developer) or determined by the Mendix developer (and then shared with the eMagiz developer).
		89	* Validate message. This determines whether the incoming message is validated before send to the microflow that handles the incoming message.
		90	* The microflow that handles the incoming message.
		91	* Auto offset reset option. This defaults to the advised setting (i.e. Latest) and is important to consider. This setting determines where the consumer group will start once it is connected for the first time to the broker.
		92
		93	== 4. Key takeaways ==
		94
		95	To listen to a topic in Mendix, you need the following:
		96
		97	* The destination (i.e., the topic from which you want to retrieve the information)
		98	* Execute microflow (the microflow that will process the data that is received in Mendix)
		99	* Optionally, you can define an import mapping and subsequent options.
		100
		101	{{warning}}To make a connection in allowing data to travel between A and B, you need to [[install>>doc:Main.eMagiz Academy.Microlearnings.Novice.Mendix Connectivity.novice-mendix-connectivity-install-emagiz-mendix-connector.WebHome\|\|target="blank"]] and do a basic [[configuration>>doc:Main.eMagiz Academy.Microlearnings.Novice.Mendix Connectivity.novice-mendix-connectivity-configure-emagiz-mendix-connector.WebHome\|\|target="blank"]] before you can set up this specific communication.{{/warning}}
		102
		103	== 5. Suggested Additional Readings ==
		104
		105	If you are interested in this topic and want more information on it, please see the following links:
		106
		107	* [[Fundamentals (Navigation)>>doc:Main.eMagiz Academy.Fundamentals.WebHome\|\|target="blank"]]
		108	** [[eMagiz Event Streaming (Explanation)>>doc:Main.eMagiz Academy.Fundamentals.fundamental-event-streaming-introduction\|\|target="blank"]]
		109	* [[eMagiz Support (Menu)>>doc:Main.eMagiz Support.WebHome\|\|target="blank"]]
		110	** [[Migration Paths (Navigation)>>doc:Main.eMagiz Support.Migration Paths.WebHome\|\|target="blank"]]
		111	*** [[Upgrade Mendix Kafka Connector to 1.3.0 (Explanation)>>doc:Main.eMagiz Support.Migration Paths.migration-path-mendix-kafka-connector-to-130.WebHome\|\|target="blank"]]
		112	* [[Crash Courses (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.WebHome\|\|target="blank"]]
		113	** [[Crash Course Event Streaming (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Event Streaming.WebHome\|\|target="blank"]]
		114	*** [[Event Catalog (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Event Streaming.crashcourse-eventstreaming-catalog\|\|target="blank"]]
		115	* [[EMagiz Connector (Search Results)>>url:https://docs.emagiz.com/bin/view/Main/Search?sort=score&sortOrder=desc&highlight=true&facet=true&r=1&f_space_facet=0%2FMain.&l_space_facet=10&f_type=DOCUMENT&f_locale=en&f_locale=&f_locale=en&text=%22emagiz+connector%22\|\|target="blank"]]
		116	* [[Kafka Explained (External)>>https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html#::text=Apache%20Kafka%20is%20a%20publish\|\|target="blank"]]
		117	* [[Topic Configuration (External)>>https://kafka.apache.org/documentation/#topicconfigs\|\|target="blank"]]
		118	* [[Kafka sizing (External)>>https://medium.com/@tsureshkumar/sizing-kafka-capacity-needed-for-your-application-fdb6f24f67cd\|\|target="blank"]]
		119	* [[Choose Partition Numbers (External)>>https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/\|\|target="blank"]]
		120	* [[Kafka Apache Introduction (External)>>https://kafka.apache.org/intro\|\|target="blank"]]
		121	)))
		122
		123	(((
		124	{{toc/}}
		125	)))
		126	{{/container}}
		127	{{/container}}

Wiki source code of Using Mendix as consumer

Navigation