eMagiz Event Streaming

{{container}}{{container layoutStyle="columns"}}(((

In this microlearning, we will introduce the essential concepts of Event Streaming. The focus will be to address the fundamental concepts of this pattern. Please refer to other Fundamentals to learn more about related items and look at the relevant microlearnings available to learn how to configure Event Streaming.

3

4

Should you have any questions, please get in touch with academy@emagiz.com.

5

6

== 1. Prerequisites ==

7

8

* Some context on Event Streaming will be helpful.

9

10

== 2. Key concepts ==

11

12

All concepts are discussed in the section below.

13

14

== 3. Introducing Event Streaming ==

15

16

Event Streaming is the concept whereby systems can produce data and consume data from a single location. In the integration world, this integration pattern is referred to as Publish-Subscribe (or PubSub). The single place is called a topic. The Publisher is often called a Producer and a Subscriber, a Consumer. The key is that a topic may contain multiple Publishers of the same data and may contain more Subscribers of the same data. Consumers are subscribed to a topic and will receive a notification that triggers a read action. The core focus of Event Streaming is high-volume and high-speed.

17

18

[[image:Main.Images.Fundamental.WebHome@fundamental-event-streaming-introduction-1.png]]

=== 3.1 Topics ===

Topics are the things where the message or data packets are stored. It is not a conventional database and uses a log-based approach to store data. Each new data packet is put on top of the stack, and an offset indicator is added, which helps consumers to understand which message is read or which ones not. The data on the topic can be spread in multiple partitions, which are the smallest technical unit where a data packet is written into. Having various partitions makes it possible to read /write data at ultra-high speed by Consumers/Producers.

23

24

Data in topics are focused on storing small events no bigger than 1 Mb per data packet/message.

25

26

[[image:Main.Images.Fundamental.WebHome@fundamental-event-streaming-introduction-2.png]]

27

28

=== 3.2 Retention ===

29

30

Retention is an essential concept that allows deleting messages from the topic in an organized manner. The key is that data should be kept on the topic for a limited amount of time or a limited amount of size.

31

32

==== 3.2.1 Retention Hours ====

33

34

Retention Hours is the number of hours data can reside on the topic before a FiFo principle of removing the first entry in the log kicks in. The moment data is still on the topic beyond this threshold; the topic will automatically start deleting the data. The default setting eMagiz provides you is 168 hours (7 days).

35

36

==== 3.2.2 Retention Bytes ====

37

38

Retention Bytes is the number of bytes available **per partition** on that topic before a FiFo principle of removing the first entry in the log kicks in. The moment your topic holds more bytes than the retention byte setting, it will automatically delete the data. The default setting eMagiz provides you is roughly 500 MB.

39

40

=== 3.3 Replication ===

41

42

The default replication mechanism is that the data will be spread across three different nodes so that no data is lost, high availability is ensured, and high performance is guaranteed. Event Streaming distributes the partitions of a particular topic across multiple brokers. By doing so, we'll get the following benefits.

43

44

* If we are to put all partitions of a topic in a single broker, the scalability of that topic will be constrained by the broker's IO throughput. A topic will never get bigger than the biggest machine in the cluster. By spreading partitions across multiple brokers, a single topic can be scaled horizontally to provide performance far beyond a single broker's ability.

45

* A single topic can be consumed by multiple consumers in parallel. Serving all partitions from a single broker limits the number of consumers it can support. Partitions on numerous brokers enable more consumers.

46

* Multiple instances of the same consumer can connect to partitions on different brokers, allowing very high message processing throughput. Each consumer instance will be served by one partition, ensuring that each record has a clear processing owner.

47

48

=== 3.4 Reading data from a topic ===

49

50

Consumers can read data from a topic at their own pace and availability. Consumers need to come and get the data. The offset of a message works as a consumer side cursor at this point. The consumer keeps track of which messages it has already consumed by keeping track of the offset of messages. After reading a message, the consumer advances its cursor to the next offset in the partition and continues. Advancing and remembering the last read offset within a partition is the responsibility of the consumer. The broker has nothing to do with it.

51

52

[[image:Main.Images.Fundamental.WebHome@fundamental-event-streaming-introduction-3.png]]

53

54

==== 3.4.1 Consumer Groups ====

55

56

Technically speaking, the Event Streaming protocol works with Consumer Groups that consist of one or more consumers. When a consumer groups holds multiple consumers each consumer within the group can be assigned to a specific partition of the topic. This is to ensure the efficiency and throughput of messages on the topic.

57

58

[[image:Main.Images.Microlearning.WebHome@crashcourse-eventstreaming-topic-and-topic-properties--topic-properties-consumer-group-explanation.png]]

59

60

The following concepts are particularly of interest when talking about consumer groups.

61

62

* All the Consumers in a group have the *same* group.id.

63

* Only one Consumer reads each partition in the topic.

64

* The maximum number of Consumers is equal to the number of partitions in the topic.

65

** If there are more consumers than partitions, then some of the consumers will remain idle.

66

* A Consumer can read from more than one partition.

67

68

=== 3.4 The Event Streaming Broker ===

69

70

The Event Streaming Broker holds the technical infrastructure to manage all these topics, Producers/Consumers associated, and many more. The Kafka framework is used inside the eMagiz Event Streaming to enable this technology piece. The entire Event Streaming Broker is part of the eMagiz platform and fully managed from the platform.

71

72

=== 3.5 Event Processing ===

73

74

eMagiz reuses the transformation capability to allow the transformation of messages. In this pattern, messages are transformed into special type flows called Event processors. The Event processor will use a topic in and a topic out, which means that data is transported from 1 topic to another. On each topic, you can define an event data model.

75

76

[[image:Main.Images.Fundamental.WebHome@fundamental-event-streaming-introduction-4.png]]

77

78

=== 3.6 Architectural components of Event Streaming in eMagiz ===

79

80

A simplified picture below is a list to illustrate the overall architecture of Event Streaming in eMagiz. The first picture is the situation where there is a multi-tenant Event Broker, and both clients are using Event Processing. Note that your solution can have producers running in the cloud and consumers on-premise or vice-versa. The lines drawn in the picture below are meant as a reference to illustrate how the event processing part works on an architectural level.

81

82

[[image:Main.Images.Fundamental.WebHome@fundamental-event-streaming-architecture-1.png]]

83

84

The picture below shows clients using the Event Broker as a passthrough (i.e. there is no event processing). So there is no need for an Event Streaming container and JMS.

85

86

[[image:Main.Images.Fundamental.WebHome@fundamental-event-streaming-architecture-2.png]]

87

88

==== 3.6.1 Event Broker ====

89

90

eMagiz is hosting an Event Broker inside the eMagiz Cloud and is accessible only via the eMagiz platform. All traffic is routed via the eMagiz platform and is protected 2-way SSL. The broker holds the specific Kafka-based technology around managing topics, Access Control Lists (ACL), users, retention, and many more. All components as described in the introduction section of this Fundamental. The broker will respond to the requests made from the eMagiz platform, such as putting a message on a topic.

91

92

==== 3.6.2 JMS and Event Streaming container ====

93

94

Where specific flows are deployed for Event Processing, the regular JMS server will control and manage the traffic. The Event Streaming Container will hold the required flows for Event Processors. The Core machine is not accessible from outside the VPC * as usual for the eMagiz core machine.

95

96

==== 3.6.3 Users ====

97

98

Users will have access to produce and consume messages. Users are managed in the User Management sections in the eMagiz Portal. Once configured, the credentials are accessible via the Event Catalog. Users can access the topics via SSL and certificates. The Key and Trust stores are available in the Event Catalog and all the required details of the Event Broker to allow access. Other access methods such as Basic Authentication or SASL like options are not supported to access the Event Broker.

99

100

== 4. Key takeaways ==

101

102

* Event Streaming provides a means for maximum decoupling of producing and consuming systems

103

* Event Streaming is an asynchronous pattern by default

104

* Event Streaming can be used for significant volume events but can als play a role in hybrid integration scenario's

105

106

== 5. Suggested Additional Readings ==

107

108

* [[Event Streaming Introduction>>https://www.emagiz.com/event-streaming/||target="blank"]]

109

* [[Crashcourse Event Streaming>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Event Streaming.WebHome||target="blank"]]

110

* [[Key Concepts Event Streaming>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Key concepts Event Streaming.WebHome||target="blank"]]

111

* [[Key Concepts Event Streaming>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Configuring Event Streaming.WebHome||target="blank"]]

112

* [[Key Concepts Event Streaming>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Event Streaming Connectors.WebHome||target="blank"]]

113

114

== 6. Silent demonstration video ==

115

116

{{video attachment="fundamental-event-streaming-introduction.mp4" reference="Main.Videos.Fundamental.WebHome"/}}

117

)))((({{toc/}}))){{/container}}{{/container}}

Wiki source code of eMagiz Event Streaming

Navigation