Changes for page Mendix -> Redshift - Data Pipeline
Last modified by Erik Bakker on 2024/02/21 21:51
From version 9.1
edited by Erik Bakker
on 2022/05/02 09:34
on 2022/05/02 09:34
Change comment:
There is no comment for this version
To version 18.1
edited by Erik Bakker
on 2022/05/02 11:10
on 2022/05/02 11:10
Change comment:
There is no comment for this version
Summary
-
Page properties (2 modified, 0 added, 0 removed)
Details
- Page properties
-
- Title
-
... ... @@ -1,1 +1,1 @@ 1 -e MagizDataSink1 +Delta Service - Data Pipeline - Content
-
... ... @@ -2,31 +2,32 @@ 2 2 {{container layoutStyle="columns"}} 3 3 ((( 4 4 == About == 5 - Withthehelpofthis functionality,you cankyourmessages into theeMagizdatasinkbucket. Once themessageisstoredin theeMagizdata sinkbucket,youwillhave theoptiontoviewthismessageinthenage phase.5 +Data pipeline configuration inside an entry connector that creates a custom Delta view on your complete table. Note that this flow only holds the logic to create a Delta based on a full load. The actual retrieval of the full-load and sending the data to another process in eMagiz are not part of the store item. 6 6 7 7 == Documentation == 8 8 9 9 ==== 1. How to use==== 10 -Once stored, eMagiz can retrieve all messages with the same ID attached during the sink action via the Manage phase. In this case, it is assumed that your eMagiz data sink is such that you can review them on the fly. Other eMagiz data sink variants are offered that required specific on-demand queries. 10 +With the help of this store item, you have a data pipeline configuration inside an entry connector that creates a custom Delta view on your complete table. This allows for the following: 11 + 12 +* Creating a delta from a full load. 11 11 12 12 ==== 2. Keynotes & restrictions==== 13 13 14 -* Note that you should think about when you want to archive your data to the eMagiz data sink. That choice determines what the impact on the functional flow is. 15 -* Don't use a wiretap to invoke this part of the flow, as that causes the sink's behavior to happen before the actual processing. If the sink fails, the message will not be delivered via the functional part of the flow. 16 -* Note that we define (among others) how the message is stored in the data sink bucket in this component. The file structure expression should not be altered, as changing this would break the functionality. 17 -* Note that the 'standard' eMagiz headers to define the source system (i.e. relevant systems) and the messageType need to be available upon sinking to make this functionality work. 16 +* The store component only provides the logic to create a delta from a full-load. 17 +* The store component does not provide the retrieval of the full load nor does it provide guidance on how to send the delta information to other parts of the solution. 18 +* Using this functionality is an expert functionality as it requires quite some context on how databases work and how eMagiz works 18 18 19 19 ==== 3. License Information==== 20 - To use this store itemyouneedto secureanadditionallicenseon the eMagiz platform.Ifyouareinterestedinsucha licensepleasecontactusatproductmanagement@emagiz.com.21 +Part of the standard license agreement as agreed upon when using the store for the first time. 21 21 22 - Wouldyouplease consulttheprivacy policy of eMagiz at thefollowing link: [[https:~~/~~/www.emagiz.com/privacy-policy/?>>url:https://www.emagiz.com/privacy-policy/?]])))23 +==== 4. Relevant eMagiz Academy Microlearnings==== 23 23 25 +[[https:~~/~~/emagiz.github.io/docs/microlearning/intermediate-data-pipelines-index>>url:https://emagiz.github.io/docs/microlearning/intermediate-data-pipelines-index]] 24 24 27 +//Would you please consult the privacy policy of eMagiz at the following link:// [[https:~~/~~/www.emagiz.com/privacy-policy/?>>url:https://www.emagiz.com/privacy-policy/?]]))) 25 25 26 -((((% border="2" cellpadding="10" cellspacing="10" style="width:292px" %) 27 -|=(% style="width: 45px;" %)#|=(% style="width: 241px;" %)Option 28 -|(% style="width:45px" %) a|(% style="width:241px" %) XML message(s) 29 -|(% style="width:45px" %) b|(% style="width:241px" %) JSON Message(s) 30 -))) 29 + 30 + 31 +((())) 31 31 {{/container}} 32 32 {{/container}}