Skip to Content
Last modified by Erik Bakker on 2024/02/21 21:51
From version 16.1
edited by Eva Torken
on 2023/08/10 13:54
Change comment: There is no comment for this version
To version 13.2
edited by eMagiz
on 2022/06/09 12:34
Change comment: Update document after refactoring.

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -Understanding Data Pipelines
1 +intermediate-data-pipelines-understanding-data-pipelines
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.etorken
1 +XWiki.marijn
Default language
... ... @@ -1,1 +1,0 @@
1 -en
Content
... ... @@ -1,12 +1,21 @@
1 -{{container}}
2 -{{container layoutStyle="columns"}}
3 -(((
1 +{{html wiki="true"}}
2 +<div class="ez-academy">
3 + <div class="ez-academy_body">
4 4  
5 +<div class="doc">
6 +
7 +
8 +
9 += Understanding Data Pipelines =
10 +
5 5  In this microlearning, we will introduce the data pipeline pattern to you. After this microlearning, you should be able to identify situations in which a data pipeline solution is the best choice for you.
6 6  With the help of such a data pipeline, you can transfer large volumes of data between a source and a sink.
7 7  
8 8  Should you have any questions, please contact academy@emagiz.com.
9 9  
16 +* Last update: April 1th 2021
17 +* Required reading time: 5 minutes
18 +
10 10  == 1. Prerequisites ==
11 11  
12 12  * Basic knowledge of the eMagiz platform
... ... @@ -19,6 +19,8 @@
19 19  
20 20  In this microlearning, we will explain a bit about what the data pipeline pattern in eMagiz means and see which choices you have for source and sink systems.
21 21  
31 +
32 +
22 22  == 3. Understanding Data Pipelines ==
23 23  
24 24  Imagine you are a global event organizer and want to track which events were hits and misses during the year.
... ... @@ -34,7 +34,7 @@
34 34  
35 35  In the flow designer of eMagiz a data pipeline looks similar to the picture shown below:
36 36  
37 -[[image:Main.Images.Microlearning.WebHome@intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png]]
48 +<p align="center">[[image:intermediate-datapipelines-understanding-data-pipelines--data-pipeline-in-emagiz.png||]]</p>
38 38  
39 39  === 3.2 Source Systems ===
40 40  
... ... @@ -52,16 +52,34 @@
52 52  * Database
53 53  * Remote directory (FTP or SFTP) in flat-file (CSV) format
54 54  
55 -== 4. Key takeaways ==
66 +===== Practice =====
56 56  
68 +== 4. Assignment ==
69 +
70 +Browse the public store of eMagiz to see which combinations of source and sink systems are used frequently and are therefore standardized.
71 +This assignment can be completed with the help of the (Academy) project that you have created/used in the previous assignment.
72 +
73 +== 5. Key takeaways ==
74 +
57 57  * A data pipeline is useful when transferring large volumes of data without the need for transformation
58 58  * Data pipelines are a standardized piece of software in eMagiz that can be implemented with ease
59 59  * eMagiz offers a limited number of source and sink options when you use the data pipeline pattern.
60 60  
61 -== 5. Suggested Additional Readings ==
62 62  
80 +
81 +== 6. Suggested Additional Readings ==
82 +
63 63  If you are interested in this topic and want more information on it please read the help text provided by eMagiz.
64 -)))
65 65  
66 -((({{toc/}}))){{/container}}
67 -{{/container}}
85 +== 7. Silent demonstration video ==
86 +
87 +This video demonstrates a working solution and how you can validate whether the refresh has worked in AWS Redshift.
88 +
89 +<iframe width="1280" height="720" src="../../vid/microlearning/intermediate-datapipelines-understanding-data-pipelines.mp4" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
90 +
91 +</div>
92 +
93 +</div>
94 +</div>
95 +
96 +{{/html}}