Skip to Content
Last modified by Erik Bakker on 2024/02/21 21:51
From version 14.1
edited by eMagiz
on 2022/06/09 12:35
Change comment: There is no comment for this version
To version 13.1
edited by eMagiz
on 2022/06/09 12:34
Change comment: Renamed from xwiki:Migrated Pages.Understanding Data Pipelines

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,0 @@
1 -Understanding Data Pipelines
Parent
... ... @@ -1,1 +1,0 @@
1 -WebHome
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.eMagiz
1 +XWiki.marijn
Content
... ... @@ -1,7 +1,13 @@
1 -{{container}}
2 -{{container layoutStyle="columns"}}
3 -(((
1 +{{html wiki="true"}}
2 +<div class="ez-academy">
3 + <div class="ez-academy_body">
4 4  
5 +<div class="doc">
6 +
7 +
8 +
9 += Understanding Data Pipelines =
10 +
5 5  In this microlearning, we will introduce the data pipeline pattern to you. After this microlearning, you should be able to identify situations in which a data pipeline solution is the best choice for you.
6 6  With the help of such a data pipeline, you can transfer large volumes of data between a source and a sink.
7 7  
... ... @@ -39,7 +39,7 @@
39 39  
40 40  In the flow designer of eMagiz a data pipeline looks similar to the picture shown below:
41 41  
42 -[[image:Main.Images.Microlearning.WebHome@intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png]]
48 +<p align="center">[[image:intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png||]]</p>
43 43  
44 44  === 3.2 Source Systems ===
45 45  
... ... @@ -56,7 +56,9 @@
56 56  * Azure Event Hub
57 57  * Database
58 58  * Remote directory (FTP or SFTP) in flat-file (CSV) format
59 -
65 +
66 +===== Practice =====
67 +
60 60  == 4. Assignment ==
61 61  
62 62  Browse the public store of eMagiz to see which combinations of source and sink systems are used frequently and are therefore standardized.
... ... @@ -78,10 +78,11 @@
78 78  
79 79  This video demonstrates a working solution and how you can validate whether the refresh has worked in AWS Redshift.
80 80  
89 +<iframe width="1280" height="720" src="../../vid/microlearning/intermediate-datapipelines-understanding-data-pipelines.mp4" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
81 81  
82 -{{video attachment="intermediate-datapipelines-understanding-data-pipelines.mp4" reference="Main.Videos.Microlearning.WebHome"/}}
91 +</div>
83 83  
84 - )))
93 +</div>
94 +</div>
85 85  
86 -((({{toc/}}))){{/container}}
87 -{{/container}}
96 +{{/html}}