Skip to Content
Last modified by Erik Bakker on 2024/02/21 21:51
From version 15.1
edited by Erik Bakker
on 2022/08/28 14:57
Change comment: There is no comment for this version
To version 13.1
edited by eMagiz
on 2022/06/09 12:34
Change comment: Renamed from xwiki:Migrated Pages.Understanding Data Pipelines

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,0 @@
1 -Understanding Data Pipelines
Parent
... ... @@ -1,1 +1,0 @@
1 -WebHome
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.ebakker
1 +XWiki.marijn
Default language
... ... @@ -1,1 +1,0 @@
1 -en
Content
... ... @@ -1,12 +1,21 @@
1 -{{container}}
2 -{{container layoutStyle="columns"}}
3 -(((
1 +{{html wiki="true"}}
2 +<div class="ez-academy">
3 + <div class="ez-academy_body">
4 4  
5 +<div class="doc">
6 +
7 +
8 +
9 += Understanding Data Pipelines =
10 +
5 5  In this microlearning, we will introduce the data pipeline pattern to you. After this microlearning, you should be able to identify situations in which a data pipeline solution is the best choice for you.
6 6  With the help of such a data pipeline, you can transfer large volumes of data between a source and a sink.
7 7  
8 8  Should you have any questions, please contact academy@emagiz.com.
9 9  
16 +* Last update: April 1th 2021
17 +* Required reading time: 5 minutes
18 +
10 10  == 1. Prerequisites ==
11 11  
12 12  * Basic knowledge of the eMagiz platform
... ... @@ -19,6 +19,8 @@
19 19  
20 20  In this microlearning, we will explain a bit about what the data pipeline pattern in eMagiz means and see which choices you have for source and sink systems.
21 21  
31 +
32 +
22 22  == 3. Understanding Data Pipelines ==
23 23  
24 24  Imagine you are a global event organizer and want to track which events were hits and misses during the year.
... ... @@ -34,7 +34,7 @@
34 34  
35 35  In the flow designer of eMagiz a data pipeline looks similar to the picture shown below:
36 36  
37 -[[image:Main.Images.Microlearning.WebHome@intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png]]
48 +<p align="center">[[image:intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png||]]</p>
38 38  
39 39  === 3.2 Source Systems ===
40 40  
... ... @@ -51,7 +51,9 @@
51 51  * Azure Event Hub
52 52  * Database
53 53  * Remote directory (FTP or SFTP) in flat-file (CSV) format
54 -
65 +
66 +===== Practice =====
67 +
55 55  == 4. Assignment ==
56 56  
57 57  Browse the public store of eMagiz to see which combinations of source and sink systems are used frequently and are therefore standardized.
... ... @@ -63,6 +63,8 @@
63 63  * Data pipelines are a standardized piece of software in eMagiz that can be implemented with ease
64 64  * eMagiz offers a limited number of source and sink options when you use the data pipeline pattern.
65 65  
79 +
80 +
66 66  == 6. Suggested Additional Readings ==
67 67  
68 68  If you are interested in this topic and want more information on it please read the help text provided by eMagiz.
... ... @@ -71,7 +71,11 @@
71 71  
72 72  This video demonstrates a working solution and how you can validate whether the refresh has worked in AWS Redshift.
73 73  
74 -{{video attachment="intermediate-datapipelines-understanding-data-pipelines.mp4" reference="Main.Videos.Microlearning.WebHome"/}})))
89 +<iframe width="1280" height="720" src="../../vid/microlearning/intermediate-datapipelines-understanding-data-pipelines.mp4" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
75 75  
76 -((({{toc/}}))){{/container}}
77 -{{/container}}
91 +</div>
92 +
93 +</div>
94 +</div>
95 +
96 +{{/html}}