Changes for page eMagiz State Generation

Summary

• Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

...	...	@@ -7,7 +7,7 @@
7	7
8	8	* Take a closer look at the Fundamentals for Messaging, API Gateway and Event Streaming
9	9
10		-== 2. Key concepts ==
	10	+== 2. Key Concepts ==
11	11
12	12	In this microlearning, the concept of state generation will be explained.
13	13
...	...	@@ -17,6 +17,8 @@
17	17
18	18	An example of a stateful application, is an application that obtains temperature measures in a room and presents a timeline of the temperature measures. A thermostat produces the temperature in this example, after which the data needs to be put into context, including the time at which a certain temperature was measured and the room in which it was measured. Then, the data can be evaluated over time, and tested against the norm. This ensures that real time actions can be generated, e.g. a fire alert can be generated when the temperature has risen by 25 degrees Celsius in 1 minute.
19	19
	20	+[[image:Main.Images.Fundamental.WebHome@fundamental-stateful-1.png]]
	21	+
20	20	== 4. Benefits and Operations ==
21	21
22	22	Benefits of real time state generation are, first of all, that data can be provided immediately and is always up-to-date. Secondly, it is possible to take real time action and take real-time decisions. Lastly, there is no need to store data unnecessarily.
...	...	@@ -36,42 +36,58 @@
36	36	Lastly, duplicate detection can be applied, meaning that duplicate data can be detected. For example, if the number of people working from home is recorded twice at a certain moment, this can be detected, and the data can be adapted accordingly.
37	37
38	38	[[image:Main.Images.Fundamental.WebHome@fundamental-stategeneration-example-duplicate-detection.png]]
39		-==== 3.2 State Operations ====
40	40
41		-Once the state store is defined, one needs different operations in order to update the state effectively.
	42	+== 5. State Generation in eMagiz ==
42	42
43		-* Retrieve - get the values of an object and the attributes
44		-* Aggregate - Increment a attribute in a state store to +1
45		-* Enrich - add new attribute of an object in the state store based on joining several streams
46		-* Transform - filter data or translate formats
47		-* Time-window - to aggregate data over time
	44	+State Generation components can be used in all eMagiz patterns. The enrich and aggregate functionalities are likely more fitting to Messaging and API Gateway as opposed to Event Streaming. For Event Streaming, the other two operations, change and duplicate detection, are more relevant.
48	48
49		-==== 3.3 State Action & networks ====
	46	+A state can most logically be created in a central place through which the data flows. The reason for this, is that such a place has all the required information flowing through its processes to generate stateful data.
50	50
51		-Once the state store has values stored, certain triggers can be defined to cause action. For instance, when the temperature is higher than 20 in the example above. These actions can be defined/modeled by the eMagiz user inside flows, whereby the evaluation of state would send a data point to the system that manages the action. For instance, the support team that manages the server park and needs to verify of the airconditioning of the server room is working properly (case high temperature).
	48	+The state store is implemented inside the eMagiz runtime using the H2 database for now. For time related operations another technology is used (Esper), and the functionality is only available on Docker based runtimes (to be release end Q2 2022). You wil find the specific Stateful components inside the Flow Designer as we use in eMagiz to model the flow. Aligned with the general concept of low-code developnment in eMagiz. For State store models, the current data modeling capabilities will be used.
52	52
53		-One can imagine that a certain action would be to update the state of another object. Or that there are several other data streams are update several states. In that way, you can imagine a complete network of data that work intertwined for you to detect rolling reality of data that can be used for data analysis and real-time decision-making.
	50	+== 6. Use Cases ==
54	54
55		-==== 3.4 Speed and throughput ====
	52	+In the following, real-life implementations are given on how eMagiz State Generation has been realized through (i) enrichment, (ii) aggregation, (iii) change detection, and (iv) duplicate detection.
56	56
57		-Many organizations hold lots of data which can be leveraged for such use cases, and often include IoT like data. Therefore, the data is usually put on event streams that are geared towards speed and throughput. In eMagiz you will find these data points usually in topics that are processed by Event Processor flows.
	54	+==== 6.1 Enrich ====
58	58
59		-==== 3.5 Pattern specifics ====
	56	+Company A works with a backend system based on ID/Name combinations. These are different for each implementation, and can be retrieved through a unique API. A difficulty that arises, is that some APIs initially only return the IDs. To prevent multiple calls to our backend system, the key/value pairs (ID/Name) need to be stored, such that these can be used in subsequent calls.
60	60
61		-At first hand it looks as if these concepts would only apply to Event Streaming. But capturing & storing a state of an object would also be handy in other patterns such as Messaging. For instance, when the order of the messages being send matters, or when there is dependency between messages that is needed to determine when a message needs to be send in time.
	58	+The eMagiz solution for Company A can be viewed below.
62	62
63		-==== 3.6 eMagiz specifics ====
	60	+[[image:Main.Images.Fundamental.WebHome@fundamental-stategeneration-usecase-enrich.png]]
64	64
65		-The state store is implemented inside the eMagiz runtime using the H2 database for now. For time related operations another technology is used (Esper), and the functionality is only available on Docker based runtimes (to be release end Q2 2022). You wil find the specific Stateful components inside the Flow Designer as we use in eMagiz to model the flow. Aligned with the general concept of low-code developnment in eMagiz. For State store models, the current data modeling capabilities will be used.
	62	+==== 6.2 Aggregation ====
66	66
67		-== 4. Key takeaways ==
	64	+Company B provides contracts and files for their customers. Their customers manually upload contracts (one by one) to their files, which may take some time. It is desired by Company B that their customers receive a push notification once their files have been uploaded successfully. However, they do not wish to spam their customers by sending a push notification after each file upload when they are uploading several files. So, the push notification should be retained for some time, and should be sent once it is (almost) certain that no additional file will be uploaded.
68	68
69		-* Stateful refers to the concept of storing a state of an object
	66	+In the figure below, the eMagiz solution for Company B can be viewed.
	67	+
	68	+[[image:Main.Images.Fundamental.WebHome@fundamental-stategeneration-usecase-aggregation.png]]
	69	+
	70	+==== 6.3 Change Detection ====
	71	+
	72	+Company C tracks packages and parcels that have been ordered by their customers. When a package is not received within its expected timeframe, an exception occurs. It can happen that such an exception resolves itself, e.g. after rendering for some time, the package is not ‘stuck’ anymore, and is being processed. Before implementing state generation, such a change was never sent to the Alert Manager, because no new exception was triggered. As a result, the employees from Company C were still working under the assumption that the exception is valid, meaning that they could still be searching for a packet that had already been received.
	73	+
	74	+With state generation, changes in package statuses are sent to the Alert Manager, so that they are aware of any (resolved) exceptions. In the figure below, the eMagiz solution for Company C can be viewed.
	75	+
	76	+[[image:Main.Images.Fundamental.WebHome@fundamental-stategeneration-usecase-change-detection.png]]
	77	+
	78	+==== 6.4 Duplicate Detection ====
	79	+
	80	+Company D is concerned with receiving meter readings. From an external system, new meter readings, or corrected meter readings are received. To protect underlying applications, it needs to be checked whether a duplicate between the external system and their own internal system exists. This can be achieved through a check for each type of meter reading. When a duplicate is found, the meter reading is marked, such that it can be checked later in the process.
	81	+The eMagiz solution for Company D can be viewed below.
	82	+
	83	+[[image:Main.Images.Fundamental.WebHome@fundamental-stategeneration-usecase-duplicate-detection.png]]
	84	+
	85	+== 4. Key Takeaways ==
	86	+
	87	+* State generation concerns the generation of a stateful application or service
	88	+ Stateful refers to the concept of storing a state of an object
70	70	* Storing a state is different compared to data in transit which is often refered as stateless data (eMagiz doesn't store or update the data when sending across)
71	71	* Stateful can be applied across all patterns in eMagiz and is embedded into the platform in such a way that it provided the same user experience
	91	+* Stateful data can be generated through four main operations: enrich, aggregation, change detection, duplicate detection
72	72
73		-[[image:Main.Images.Fundamental.WebHome@fundamental-stateful-1.png]]
74		-
75	75	== 5. Suggested Additional Readings ==
76	76
77	77	N/A