Skip to Content
Last modified by Erik Bakker on 2024/02/21 21:51
From version 13.2
edited by eMagiz
on 2022/06/09 12:34
Change comment: Update document after refactoring.
To version 16.1
edited by Eva Torken
on 2023/08/10 13:54
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -intermediate-data-pipelines-understanding-data-pipelines
1 +Understanding Data Pipelines
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.marijn
1 +XWiki.etorken
Default language
... ... @@ -1,0 +1,1 @@
1 +en
Content
... ... @@ -1,21 +1,12 @@
1 -{{html wiki="true"}}
2 -<div class="ez-academy">
3 - <div class="ez-academy_body">
1 +{{container}}
2 +{{container layoutStyle="columns"}}
3 +(((
4 4  
5 -<div class="doc">
6 -
7 -
8 -
9 -= Understanding Data Pipelines =
10 -
11 11  In this microlearning, we will introduce the data pipeline pattern to you. After this microlearning, you should be able to identify situations in which a data pipeline solution is the best choice for you.
12 12  With the help of such a data pipeline, you can transfer large volumes of data between a source and a sink.
13 13  
14 14  Should you have any questions, please contact academy@emagiz.com.
15 15  
16 -* Last update: April 1th 2021
17 -* Required reading time: 5 minutes
18 -
19 19  == 1. Prerequisites ==
20 20  
21 21  * Basic knowledge of the eMagiz platform
... ... @@ -28,8 +28,6 @@
28 28  
29 29  In this microlearning, we will explain a bit about what the data pipeline pattern in eMagiz means and see which choices you have for source and sink systems.
30 30  
31 -
32 -
33 33  == 3. Understanding Data Pipelines ==
34 34  
35 35  Imagine you are a global event organizer and want to track which events were hits and misses during the year.
... ... @@ -45,7 +45,7 @@
45 45  
46 46  In the flow designer of eMagiz a data pipeline looks similar to the picture shown below:
47 47  
48 -<p align="center">[[image:intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png||]]</p>
37 +[[image:Main.Images.Microlearning.WebHome@intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png]]
49 49  
50 50  === 3.2 Source Systems ===
51 51  
... ... @@ -63,34 +63,16 @@
63 63  * Database
64 64  * Remote directory (FTP or SFTP) in flat-file (CSV) format
65 65  
66 -===== Practice =====
55 +== 4. Key takeaways ==
67 67  
68 -== 4. Assignment ==
69 -
70 -Browse the public store of eMagiz to see which combinations of source and sink systems are used frequently and are therefore standardized.
71 -This assignment can be completed with the help of the (Academy) project that you have created/used in the previous assignment.
72 -
73 -== 5. Key takeaways ==
74 -
75 75  * A data pipeline is useful when transferring large volumes of data without the need for transformation
76 76  * Data pipelines are a standardized piece of software in eMagiz that can be implemented with ease
77 77  * eMagiz offers a limited number of source and sink options when you use the data pipeline pattern.
78 78  
61 +== 5. Suggested Additional Readings ==
79 79  
80 -
81 -== 6. Suggested Additional Readings ==
82 -
83 83  If you are interested in this topic and want more information on it please read the help text provided by eMagiz.
64 +)))
84 84  
85 -== 7. Silent demonstration video ==
86 -
87 -This video demonstrates a working solution and how you can validate whether the refresh has worked in AWS Redshift.
88 -
89 -<iframe width="1280" height="720" src="../../vid/microlearning/intermediate-datapipelines-understanding-data-pipelines.mp4" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
90 -
91 -</div>
92 -
93 -</div>
94 -</div>
95 -
96 -{{/html}}
66 +((({{toc/}}))){{/container}}
67 +{{/container}}