Last modified by Danniar Firdausy on 2023/08/11 10:54
From version 38.2
edited by Danniar Firdausy
on 2023/08/11 10:49
Change comment: There is no comment for this version
To version 21.2
edited by eMagiz
on 2022/06/08 12:30
Change comment: Update document after refactoring.

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -Deciding upon Integration Pattern
1 +intermediate-discover-your-integration-landscape-determining-integration-pattern
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.dfirdausy
1 +XWiki.eMagiz
Default language
... ... @@ -1,1 +1,0 @@
1 -en
Content
... ... @@ -1,38 +1,113 @@
1 1  {{container}}
2 2  {{container layoutStyle="columns"}}
3 3  (((
4 -In the previous microlearning, you have learned the three "first-class" integration patterns provided by eMagiz. With this exercise, we will provide you with a business case to reflect on which of these patterns are best suited for the case.
5 -
6 -Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
4 +In eMagiz, we differentiate between three "first-class" integration patterns. Each of these patterns is best suited for particular integration challenges. In this microlearning, we will see how to compare the three integration patterns to decide which pattern to choose.
7 7  
6 +Should you have any questions, please get in touch with academy@emagiz.com.
7 +
8 +* Last update: August 5th, 2021
9 +* Required reading time: 7 minutes
10 +
8 8  == 1. Prerequisites ==
9 9  
10 -* Have read the “Intermediate - Discover your integration landscape - Determining Integration Pattern” microlearning.
13 +* Basic knowledge of the eMagiz platform
11 11  
12 -== 2. Exercise ==
15 +== 2. Key concepts ==
13 13  
14 -Below, the exercise for this part of the intermediate course, which centers around deciding upon integration patterns suitable for a business case, is given. By integration patterns, we mean the three "first-class" patterns that are supported by eMagiz (i.e., Messaging, Event Streaming, and API Gateway). Note that each pattern has its own pros and cons. Therefore, decide which pattern suits best based on the technical checks and balances.
17 +This microlearning centers on determining integration patterns.
15 15  
16 -===2.1 Business Case===
17 -**Read and understand the following business case**
18 -
19 -ABC Logistiek B.V. is a medium-sized logistics provider offering land transport services in The Netherlands. For regulation compliance purposes, they are required to report their transport trip data to the Dutch Central Bureau of Statistics (CBS). The CBS requires this data from logistics providers across the country for the purpose of tracking national logistics performance. Logistics providers normally report their transport trip in batches through SFTP connection at 3 monthly intervals and, thus, does not happen on a daily basis. Different logistics providers may have different mileage, but ABC Logiestiek B.V. in particular sends 2-3 GBs of transport trip data every batch. At the same time, while they have to report to CBS, they are also communicating with their partners and customers to keep their business running. In this case, they are tracking and monitoring the location of the trucks carrying the goods, in which real-time data (-+ 15KBs data, every 30 seconds) is captured and transmitted to both their own and partners’ enterprise systems.
20 -
21 -Logistics providers normally have different ways of working with their data and enterprise systems in their day-to-day operations. CBS is aware of this, and thus they introduced a common data model (see OTM as an example) for logistics providers to adhere to. Harmonizing this transport trip messages with other players also unlocks the possibility of easier communication and coordination with other partners in the network as well. As promising as this initiative seems, ABC Logiestiek B.V. is in fact using several enterprise systems (e.g., TMS, CRM, IoT systems, etc.) that mostly use proprietary data schema, message definitions, and communication protocols that do not have one-to-one relation with OTM. This makes them adjusting to their supply chain network challenging. Manual filling, copying, and pasting data from one system to another is no longer an option due to the amount of work required. Establishing custom one-to-one integrations is also indeed an option. But, this will impose steep operational costs, complexity, and risks in the long run for them. Therefore, the company approaches eMagiz to tackle this issue as eMagiz supports system integrations, message transformation, and routings through different options of integration patterns.
22 -
23 -As a Business IT Consultant/Integration Specialist, you are to provide ABC Logistiek B.V. with a solution for their integration landscape. Particularly, how to approach their problem and which integration pattern(s) eMagiz offers that are suitable for their case.
19 +The focal point of this microlearning will be to figure out how you can best decide between the various "first-class" integration patterns.
24 24  
25 -===2.2 Questions to Ask Yourself ===
21 +* The key aspects are:
22 + * The three "first-class" patterns are messaging, event streaming, and API gateway
23 + * Each pattern has pros and cons
24 + * Decide which pattern suits best based on business and technical checks and balances
26 26  
27 -Based on the business case described so far:
28 -* What will be the integration pattern(s) supported by eMagiz suitable for them?
29 -* Which functionality can be solved with the selected integration patterns?
30 -* To propose a better solution for this business case, what other questions should be explored and answered?
31 -* Can we make a decision tree/flowchart for this case based on what we know so far about the eMagiz Platform? For this, please check out: “Intermediate - Discover your integration landscape” course
32 32  
33 -== 3. Solution ==
34 34  
35 -Discuss internally within your team what will be the suitable solution, and why you think that A or B is the best solution.
28 +== 3. Determining Integration Pattern ==
29 +
30 +In eMagiz, we differentiate between three "first-class" integration patterns. Each of these patterns is best suited for particular integration challenges. In this microlearning, we will see how to compare the three integration patterns to decide which pattern to choose.
31 +
32 +The focal point of this microlearning will be to figure out how you can best decide between the various "first-class" integration patterns.
33 +
34 +* The key aspects are:
35 + * The three "first-class" patterns are messaging, event streaming, and API gateway
36 + * Each pattern has pros and cons
37 + * Decide which pattern suits best based on business and technical checks and balances
38 +
39 +As stated above, there are three "first-class" integration patterns within eMagiz. Each of these patterns has a unique way of processing the data. In this microlearning, we will look at how to decide which pattern best fits your integration challenge. Before we do that, let us first quickly look at the three integration patterns. In the picture below, you see a visual representation of each of the three patterns.
40 +
41 +[[image:Main.Images.Microlearning.WebHome@intermediate-discover-your-integration-landscape-determining-integration-pattern--the-three-patterns-visualized.png]]
42 +
43 +As you can see, each pattern works differently. If you want an in-depth introduction to messaging, please check out this [microlearning](crashcourse-messaging-introduction.md). If you want an in-depth introduction to event streaming, please check out this [microlearning](crashcourse-eventstreaming-event-streaming-introduction.md). If you want an in-depth introduction to API gateway, please check out this [microlearning](crashcourse-api-gateway-introduction.md).
44 +
45 +Because each pattern works differently, each pattern has its pros and its cons. Depending on your specific use case, the balance between the pros and cons of one of the patterns will favor the pros. As a result, you should choose that option. But then, you ask, on what grounds should I make the decision? The answer is a series of checks and balances that look both at the business and technical aspects.
46 +
47 +=== 3.1 Checks and balances ===
48 +
49 +At first, look at the business side of your integration scenario before delving into the technical disqualifiers. You should ask the first question: **Does the business process require a real-time response (for both success and error messages)?**. This question inevitably leads to the following question: **What is required?** In this case, required means that a person or system needs to wait for the answer (good or bad) before continuing with their process. If the answer is irrelevant for the progress of the process, real-time response is **not** required.
50 +
51 +Based on your response to this first question, you have already made a distinctive choice that eliminates various possibilities. With that in mind, we will continue our quest in two 'lanes'. We start with asynchronous, and after that, we come back to synchronous.
52 +
53 +==== 3.1.1 Asynchronous ====
54 +
55 +Now that we have determined that our integration should run asynchronously, it becomes time to ask ourselves a second question: **With what frequency is the data delivered to eMagiz?**. If there is a high throughput requirement, the choice might differ when you only receive data once. You should ask yourself directly following this question: **How much data do I expect to process within one specific time frame?**. So, if the data is delivered five times a day, it will become relevant to determine how much data is produced each time. If the volume becomes vast in terms of numbers or size, selecting a particular pattern might change. As a rule of thumb, if you have low-frequency low throughput traffic **messaging** will be a solid choice. If you have high-frequency high throughput traffic, **event streaming** looks to be the best candidate at this point.
56 +
57 +We are not done yet with our investigation. The next part of our investigation is about filtering and availability. A good question to ask yourself is: **Do all receiving systems want to receive the same dataset, or will this differ?**. The answer to this question will tell you something about the solution. If you opted for **event streaming** in this case, that would mean that you need to filter between topics to ensure that each consuming party gets the correct data set. This choice automatically implies that you need some additional storage capacity to make this happen.
58 +
59 +For **messaging**, this would mean filtering per offramp towards a system. The second question is on availability: **Are the target systems always online?** With availability, we mean 99,5% or more. When systems are not online or are hindered in their core process by data coming in at all hours of the day, you might want to retain data for a certain amount of time and let the consuming application consume the data when they want. If so **event streaming** will become more and more interesting over **messaging**.
60 +
61 +Now that we are in the retention area, the question becomes how long data needs to be retained. Therefore we need to ask ourselves: **Does the data need to be retained for less than seven days?** If the answer is yes, we need to ask ourselves a follow-up question. **Are we allowed to keep this data under the compliance rules of the company?**. If the answer to that question is no, you cannot use event streaming.
62 +As a consequence, the best pattern, in this case, will be messaging. Note that data is not retained within the **messaging** pattern but can be redelivered in case of emergency (i.e., when the system is down that one hour per year). If eMagiz can (temporarily) store the data, we will need to look at the technical aspect of the connected systems. The question then becomes: **Can the systems directly connect to a Kafka cluster?**. If so **event streaming** is your best choice. If not, you should consider a **hybrid case of messaging and event streaming** or go **complete messaging** is the frequency and throughput is shallow.
63 +
64 +But then what if I have to retain the data for a period longer than seven days? In those cases, you will need an extension on the pattern to store data in S3 or another storage facility. Depending on what needs to be stored, eMagiz offers specific standardized store components to help you with this.
65 +
66 +==== 3.1.2 Synchronous ====
67 +
68 +Now that we have determined that our integration should run synchronously, we should first ask ourselves: **Can the data provider give back a timely response?** With timely in this context, we mean within 20 seconds. If the answer is yes, the **API gateway** pattern becomes the front runner. If not, you should see what kind of response the system calls your integration solution needs to continue. Is this simply an acknowledgment, or does the response need to contain specific data? If the latter is the case, you have a mismatch between the way both systems operate and a challenge on your hand. In those cases, the **messaging** pattern should be utilized. Note that the integration alone cannot be synchronous anymore but should be set up asynchronously to remove the time constraint on the response.
69 +
70 +Assuming the data provider can respond promptly, we need to continue our investigations. As the **API gateway** requires that clients calling the API gateway call it via a REST interface, the next question becomes: **Can the client calling the API Gateway to communicate via the REST protocol?**. If the answer is yes, the **API gateway** is the preferred option. If the answer is no, you should select the **messaging** pattern. This time, however, the integration should be synchronous in nature.
71 +
72 +Another aspect to consider is user management. The **API gateway** pattern comes with build-in user management functionality, whereas the **messaging** does not. So you should ask yourself the question: **Is centralized API Access required**? If so, the **API gateway** pattern best suits your needs. If not, you should base your decision on information gather before.
73 +
74 +The last consideration we need to make concerns security. The **API gateway** pattern is set up so that all clients need to provide the same type of authentication. So, for example, when you have secured your API Gateway via the OAuth2.0 protocol, all clients that want to call your API Gateway should authorize themselves via this protocol. This could be a technical disqualifier for a particular integration solution. So the question becomes: **Can the client provide the proper authorization that we require?**. Note that when it is your first **API gateway** implementation, you might want to discuss the alternatives to agree upon a security protocol that best suits all parties involved.
75 +
76 +Taking this all into consideration should guide you to select a particular integration pattern based on various business and technical checks and balances.
77 +
78 +=== 3.2 Licensing, Implementation Costs, and Resource Costs ===
79 +
80 +Apart from the above considerations, there are also costs involved in making the decision. Unlocking a specific pattern will incur a licensing fee. So only using a particular pattern for only one integration solution might be less attractive. On the other hand, each pattern has differing implementation costs within eMagiz. Below you will see the patterns ranked based on the implementation costs (low to high):
81 +
82 +* Event Streaming
83 +* API Gateway
84 +* Messaging
85 +
86 +So that means that you might save money in the short with sticking what you know but could end up paying more due to (significantly) higher implementation costs. Another piece of the puzzle is the resource costs. For this aspect also the resource costs differ per integration pattern. Here we see that taking a messaging solution would incur the highest resource costs. API Gateway and Event Streaming roughly take up the same amount of resource costs. We want to note that a large portion of the resource costs for Event Streaming lies in storage. So when you can reduce that component, the Event Streaming option would become even more cost-friendly. Hopefully, this microlearning will help you to make the correct decision per integration question.
87 +
88 +===== Practice =====
89 +
90 +== 4. Assignment ==
91 +
92 +Reflect on the choices made within various projects and see if you would change the specific implementation with what you know now.
93 +
94 +== 5. Key takeaways ==
95 +
96 +* The key aspects are:
97 + * The three "first-class" patterns are messaging, event streaming, and API gateway
98 + * Each pattern has pros and cons
99 + * Decide which pattern suits best based on business and technical checks and balances
100 +
101 +
102 +
103 +== 6. Suggested Additional Readings ==
104 +
105 +If you are interested in this topic, please get in touch with academy@emagiz.com.
106 +
107 +== 7. Silent demonstration video ==
108 +
109 +As this is a more theoretical microlearning, we have no video for this.
110 +
36 36  )))
37 37  
38 38  ((({{toc/}}))){{/container}}