eMagiz State Generation

In this fundamental, we'll take a look at the capability of eMagiz around storing a state of a data packet of message send across the platform.

Should you have any questions, please get in touch with academy@emagiz.com.

1. Prerequisites

• Take a closer look at the Fundamentals for Messaging, API Gateway and Event Streaming

2. Key concepts

In this microlearning, the concept of state generation will be explained.

3. Definition

State generation concerns the generation of a stateful application or service. This means that, for the application or service to function, data is stored and past states are used to derive information. Such a state may e.g. be a temperature or heartrate measure. Stateful applications or services are opposed to stateless applications or services, which do not require data to function and merely represent the current state.

An example of a stateful application, is an application that obtains temperature measures in a room and presents a timeline of the temperature measures. A thermostat produces the temperature in this example, after which the data needs to be put into context, including the time at which a certain temperature was measured and the room in which it was measured. Then, the data can be evaluated over time, and tested against the norm. This ensures that real time actions can be generated, e.g. a fire alert can be generated when the temperature has risen by 25 degrees Celsius in 1 minute.

4. Benefits and Operations

Benefits of real time state generation are, first of all, that data can be provided immediately and is always up-to-date. Secondly, it is possible to take real time action and take real-time decisions. Lastly, there is no need to store data unnecessarily.

Stateful data can be generated through four main operations. First of all, through enriching data. This means that information should be added to the data, so that it has more meaning. This can be achieved by storing states in a database. For example, when it is known that, if a person works from home, this is stored as A, and if a person works in the office, this is stored as B, the data can be enriched with ‘Home’ and ‘Office’. To ensure this, a database should be present storing the connection between A and Home, and B and Office.

Secondly, stateful data can be obtained through aggregating data. Aggregation concerns a computation over a certain range of time, such as an average, minimum or maximum. For example, when the number of people working from home, or working at the office is stored, the average number of people working at home or at the office during the past 30 minutes can be determined.

Thirdly, change detection can be applied. This means that a change in the data can be detected. For example, when the temperature at home rises to more than 30 degrees, this can be detected. It can then be defined that, when the temperature at home is more than 30 degrees and the heating is on, the heating should be turned off. In this example, the data is enriched with the last state of the air conditioning.

Lastly, duplicate detection can be applied, meaning that duplicate data can be detected. For example, if the number of people working from home is recorded twice at a certain moment, this can be detected, and the data can be adapted accordingly.

3.2 State Operations

Once the state store is defined, one needs different operations in order to update the state effectively.

• Retrieve - get the values of an object and the attributes
• Aggregate - Increment a attribute in a state store to +1
• Enrich - add new attribute of an object in the state store based on joining several streams
• Transform - filter data or translate formats
• Time-window - to aggregate data over time

3.3 State Action & networks

Once the state store has values stored, certain triggers can be defined to cause action. For instance, when the temperature is higher than 20 in the example above. These actions can be defined/modeled by the eMagiz user inside flows, whereby the evaluation of state would send a data point to the system that manages the action. For instance, the support team that manages the server park and needs to verify of the airconditioning of the server room is working properly (case high temperature).

One can imagine that a certain action would be to update the state of another object. Or that there are several other data streams are update several states. In that way, you can imagine a complete network of data that work intertwined for you to detect rolling reality of data that can be used for data analysis and real-time decision-making.

3.4 Speed and throughput

Many organizations hold lots of data which can be leveraged for such use cases, and often include IoT like data. Therefore, the data is usually put on event streams that are geared towards speed and throughput. In eMagiz you will find these data points usually in topics that are processed by Event Processor flows.

3.5 Pattern specifics

At first hand it looks as if these concepts would only apply to Event Streaming. But capturing & storing a state of an object would also be handy in other patterns such as Messaging. For instance, when the order of the messages being send matters, or when there is dependency between messages that is needed to determine when a message needs to be send in time.

3.6 eMagiz specifics

The state store is implemented inside the eMagiz runtime using the H2 database for now. For time related operations another technology is used (Esper), and the functionality is only available on Docker based runtimes (to be release end Q2 2022). You wil find the specific Stateful components inside the Flow Designer as we use in eMagiz to model the flow. Aligned with the general concept of low-code developnment in eMagiz. For State store models, the current data modeling capabilities will be used.

4. Key takeaways

• Stateful refers to the concept of storing a state of an object
• Storing a state is different compared to data in transit which is often refered as stateless data (eMagiz doesn't store or update the data when sending across)
• Stateful can be applied across all patterns in eMagiz and is embedded into the platform in such a way that it provided the same user experience

5. Suggested Additional Readings

N/A

6. Silent demonstration video

N/A