Skip to Content
Last modified by Erik Bakker on 2024/02/21 21:51
From version 13.2
edited by eMagiz
on 2022/06/09 12:34
Change comment: Update document after refactoring.
To version 15.1
edited by Erik Bakker
on 2022/08/28 14:57
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -intermediate-data-pipelines-understanding-data-pipelines
1 +Understanding Data Pipelines
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.marijn
1 +XWiki.ebakker
Default language
... ... @@ -1,0 +1,1 @@
1 +en
Content
... ... @@ -1,21 +1,12 @@
1 -{{html wiki="true"}}
2 -<div class="ez-academy">
3 - <div class="ez-academy_body">
1 +{{container}}
2 +{{container layoutStyle="columns"}}
3 +(((
4 4  
5 -<div class="doc">
6 -
7 -
8 -
9 -= Understanding Data Pipelines =
10 -
11 11  In this microlearning, we will introduce the data pipeline pattern to you. After this microlearning, you should be able to identify situations in which a data pipeline solution is the best choice for you.
12 12  With the help of such a data pipeline, you can transfer large volumes of data between a source and a sink.
13 13  
14 14  Should you have any questions, please contact academy@emagiz.com.
15 15  
16 -* Last update: April 1th 2021
17 -* Required reading time: 5 minutes
18 -
19 19  == 1. Prerequisites ==
20 20  
21 21  * Basic knowledge of the eMagiz platform
... ... @@ -28,8 +28,6 @@
28 28  
29 29  In this microlearning, we will explain a bit about what the data pipeline pattern in eMagiz means and see which choices you have for source and sink systems.
30 30  
31 -
32 -
33 33  == 3. Understanding Data Pipelines ==
34 34  
35 35  Imagine you are a global event organizer and want to track which events were hits and misses during the year.
... ... @@ -45,7 +45,7 @@
45 45  
46 46  In the flow designer of eMagiz a data pipeline looks similar to the picture shown below:
47 47  
48 -<p align="center">[[image:intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png||]]</p>
37 +[[image:Main.Images.Microlearning.WebHome@intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png]]
49 49  
50 50  === 3.2 Source Systems ===
51 51  
... ... @@ -62,9 +62,7 @@
62 62  * Azure Event Hub
63 63  * Database
64 64  * Remote directory (FTP or SFTP) in flat-file (CSV) format
65 -
66 -===== Practice =====
67 -
54 +
68 68  == 4. Assignment ==
69 69  
70 70  Browse the public store of eMagiz to see which combinations of source and sink systems are used frequently and are therefore standardized.
... ... @@ -76,8 +76,6 @@
76 76  * Data pipelines are a standardized piece of software in eMagiz that can be implemented with ease
77 77  * eMagiz offers a limited number of source and sink options when you use the data pipeline pattern.
78 78  
79 -
80 -
81 81  == 6. Suggested Additional Readings ==
82 82  
83 83  If you are interested in this topic and want more information on it please read the help text provided by eMagiz.
... ... @@ -86,11 +86,7 @@
86 86  
87 87  This video demonstrates a working solution and how you can validate whether the refresh has worked in AWS Redshift.
88 88  
89 -<iframe width="1280" height="720" src="../../vid/microlearning/intermediate-datapipelines-understanding-data-pipelines.mp4" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
74 +{{video attachment="intermediate-datapipelines-understanding-data-pipelines.mp4" reference="Main.Videos.Microlearning.WebHome"/}})))
90 90  
91 -</div>
92 -
93 -</div>
94 -</div>
95 -
96 -{{/html}}
76 +((({{toc/}}))){{/container}}
77 +{{/container}}