Changes for page Mendix -> Redshift - Data Pipeline
Last modified by Erik Bakker on 2024/02/21 21:51
From version 20.1
edited by Erik Bakker
on 2022/05/02 11:40
on 2022/05/02 11:40
Change comment:
There is no comment for this version
To version 18.1
edited by Erik Bakker
on 2022/05/02 11:10
on 2022/05/02 11:10
Change comment:
There is no comment for this version
Summary
-
Page properties (2 modified, 0 added, 0 removed)
Details
- Page properties
-
- Title
-
... ... @@ -1,1 +1,1 @@ 1 - Local File-> RemoteFile - Data Pipeline1 +Delta Service - Data Pipeline - Content
-
... ... @@ -2,29 +2,20 @@ 2 2 {{container layoutStyle="columns"}} 3 3 ((( 4 4 == About == 5 -Data pipeline configuration inside an entry connector t otransferlocalCVS filesCSVfilesonSFTPserver.This flowcanalsobeused asstartingpoint forotherdatapipeline integrations.5 +Data pipeline configuration inside an entry connector that creates a custom Delta view on your complete table. Note that this flow only holds the logic to create a Delta based on a full load. The actual retrieval of the full-load and sending the data to another process in eMagiz are not part of the store item. 6 6 7 7 == Documentation == 8 8 9 9 ==== 1. How to use==== 10 -With the help of this store item, you have a data pipeline configuration inside an entry connector t otransferlocalCSV filesCSVfileson theSFTP server. This allows for the following:10 +With the help of this store item, you have a data pipeline configuration inside an entry connector that creates a custom Delta view on your complete table. This allows for the following: 11 11 12 -* Reading CSV data from a local file system. 13 -* Convert data to flat files (CSV). 14 -* Writing data to SFTP. 15 -* Job dashboard (See Deploy --> Runtime Dashboard --> [Select Runtime] - > [Select Flow] -> Jobs ). 12 +* Creating a delta from a full load. 16 16 17 17 ==== 2. Keynotes & restrictions==== 18 18 19 -* Source data must be shared in Mendix using a 'Published OData service'. If applicable in the settings tab set the association configuration to 'As an associated object id'. 20 -* Mendix application must be reachable by the eMagiz runtime. 21 -* Before use, please rename job 'system.message.job' and step 'system.message.step1' to match your system and integration name. Make sure to update support.job-launch-request accordingly. 22 -* When using multiple pipeline flows on one runtime, an h2 'already in use' exception might occur. To prevent these errors please keep the name of the h2 database different for each flow on one runtime by renaming the 'dp.h2.message.database' deploy property in the "support.h2-database" component. 23 -* When you want to import the flow multiple times for one system, you should change the names of the following properties by replacing 'message' with the name of the message type of each integration: 24 - ** dp.odata.message.cron (component: receive.cron) 25 - ** dp.odata.message.url (component: system.message.job -> system.message.step1 -> Item reader) 26 - ** dp.h2.message.database (component: support.h2-database) 27 - ** dp.sftp.message.filename.prefix (component: transform.job-launch-request - > Expression) 16 +* The store component only provides the logic to create a delta from a full-load. 17 +* The store component does not provide the retrieval of the full load nor does it provide guidance on how to send the delta information to other parts of the solution. 18 +* Using this functionality is an expert functionality as it requires quite some context on how databases work and how eMagiz works 28 28 29 29 ==== 3. License Information==== 30 30 Part of the standard license agreement as agreed upon when using the store for the first time.