Changes for page Mendix -> Redshift - Data Pipeline
Last modified by Erik Bakker on 2024/02/21 21:51
From version 8.1
edited by Erik Bakker
on 2022/05/02 09:31
on 2022/05/02 09:31
Change comment:
There is no comment for this version
To version 19.1
edited by Erik Bakker
on 2022/05/02 11:39
on 2022/05/02 11:39
Change comment:
There is no comment for this version
Summary
-
Page properties (2 modified, 0 added, 0 removed)
Details
- Page properties
-
- Title
-
... ... @@ -1,1 +1,1 @@ 1 - SOAPWebserviceConnectivity1 +Local File -> Remote File - Data Pipeline - Content
-
... ... @@ -2,40 +2,41 @@ 2 2 {{container layoutStyle="columns"}} 3 3 ((( 4 4 == About == 5 - This storeelementwill allow youcreatean easy connectiontoSOAP basedwebservices.XMLdatacan be sendto thesewebservices,andforeach connectivity type thereisaFlow Fragmentvailable.5 +Data pipeline configuration inside an entry connector to transfer local CVS files to CSV files on SFTP server. This flow can also be used as starting point for other data pipeline integrations. 6 6 7 7 == Documentation == 8 8 9 9 ==== 1. How to use==== 10 -Ensure to locate the right connectivity method and import the right flow fragment. 10 +With the help of this store item, you have a data pipeline configuration inside an entry connector to transfer local CSV files to CSV files on the SFTP server. This allows for the following: 11 + 12 +* Reading CSV data from a local file system. 13 +* Convert data to flat files (CSV). 14 +* Writing data to SFTP. 15 +* Job dashboard (See Deploy --> Runtime Dashboard --> [Select Runtime] - > [Select Flow] -> Jobs ). 11 11 12 12 ==== 2. Keynotes & restrictions==== 13 13 14 -* These flow fragments will contain only the eMagiz components to make a call to a SOAP based webservice. 15 -* The basic authentication flow fragment requires empty key and trust store that are to be added separately. These are not included in this store component. 19 +* Source data must be shared in Mendix using a 'Published OData service'. If applicable in the settings tab set the association configuration to 'As an associated object id'. 20 +* Mendix application must be reachable by the eMagiz runtime. 21 +* Before use, please rename job 'system.message.job' and step 'system.message.step1' to match your system and integration name. Make sure to update support.job-launch-request accordingly. 22 +* When using multiple pipeline flows on one runtime, an h2 'already in use' exception might occur. To prevent these errors please keep the name of the h2 database different for each flow on one runtime by renaming the 'dp.h2.message.database' deploy property in the "support.h2-database" component. 23 +* When you want to import the flow multiple times for one system, you should change the names of the following properties by replacing 'message' with the name of the message type of each integration: 24 + * dp.odata.message.cron (component: receive.cron) 25 + * dp.odata.message.url (component: system.message.job -> system.message.step1 -> Item reader) 26 + * dp.h2.message.database (component: support.h2-database) 27 + * dp.sftp.message.filename.prefix (component: transform.job-launch-request - > Expression) 16 16 17 17 ==== 3. License Information==== 18 18 Part of the standard license agreement as agreed upon when using the store for the first time. 19 19 20 20 ==== 4. Relevant eMagiz Academy Microlearnings==== 21 -[[https:~~/~~/emagiz.github.io/docs/microlearning/novice-soap-webservice-connectivity-authorization-calling-a-soap-webservice>>url:https://emagiz.github.io/docs/microlearning/novice-soap-webservice-connectivity-authorization-calling-a-soap-webservice]] 22 22 34 +[[https:~~/~~/emagiz.github.io/docs/microlearning/intermediate-data-pipelines-index>>url:https://emagiz.github.io/docs/microlearning/intermediate-data-pipelines-index]] 23 23 24 -Would you please consult the privacy policy of eMagiz at the following link: [[https:~~/~~/www.emagiz.com/privacy-policy/?>>url:https://www.emagiz.com/privacy-policy/?]]))) 36 +//Would you please consult the privacy policy of eMagiz at the following link:// [[https:~~/~~/www.emagiz.com/privacy-policy/?>>url:https://www.emagiz.com/privacy-policy/?]]))) 25 25 26 26 27 27 28 -((((% border="2" cellpadding="10" cellspacing="10" style="width:292px" %) 29 -|=(% style="width: 45px;" %)#|=(% style="width: 241px;" %)Option 30 -|(% style="width:45px" %) a|(% style="width:241px" %) Basic Authentication 31 -|(% style="width:45px" %) b|(% style="width:241px" %) WS Addressing - Call back 32 -|(% style="width:45px" %) c|(% style="width:241px" %) Client certificate 33 -|(% style="width:45px" %) d|(% style="width:241px" %) Mapped headers 34 -|(% style="width:45px" %) e|(% style="width:241px" %) Token based - static 35 -|(% style="width:45px" %) f|(% style="width:241px" %) Token based - dynamic 36 -|(% style="width:45px" %) g|(% style="width:241px" %) SSL 37 -|(% style="width:45px" %) h|(% style="width:241px" %) No authentication 38 -|(% style="width:45px" %) i|(% style="width:241px" %) Dynamic URL 39 -))) 40 +((())) 40 40 {{/container}} 41 41 {{/container}}