Skip to Content
Last modified by Erik Bakker on 2024/02/21 21:51
From version 5.1
edited by Erik Bakker
on 2022/05/02 08:20
Change comment: There is no comment for this version
To version 29.1
edited by Erik Bakker
on 2024/02/21 21:51
Change comment: Renamed back-links.

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -SOAP Web service Connectivity
1 +Mendix -> Redshift - Data Pipeline
Content
... ... @@ -1,40 +1,52 @@
1 1  {{container}}
2 2  {{container layoutStyle="columns"}}
3 3  (((
4 -= About =
5 -This store element will allow you to create an easy connection to SOAP based web services. XML data can be send to these web services, and for each connectivity type there is a Flow Fragment available.
4 +== About ==
6 6  
7 -= Documentation =
6 +Data pipeline configuration inside an entry connector to transfer Mendix App object data to an Amazon Redshift table. This flow can also be used as starting point for other data pipeline integrations.
8 8  
9 - **~1. How to use**
10 -Ensure to locate the right connectivity method and import the right flow fragment.
8 +== Documentation ==
11 11  
12 -**2. Keynotes & restrictions **
10 +==== 1. How to use ====
13 13  
14 -a. These flow fragments will contain only the eMagiz components to make a call out to the SOAP based webservice - These components do not hold any system messages - The basic authentication flow fragment requires empty key and trust store that are to be added separately. These are not included in this store component.
12 +With the help of this store item, you have a data pipeline configuration inside an entry connector to transfer Mendix App object data to an Amazon Redshift table. This allows for the following:
15 15  
16 -**3. License Information**
14 +* Reading data from Mendix using OData v3.
15 +* Writing data to [[Amazon Redshift>>https://docs.aws.amazon.com/redshift/latest/dg/t_loading-tables-from-s3.html||target="blank"]] using Amazon S3.
16 +* Merge behavior using [[staging tables>>https://docs.aws.amazon.com/redshift/latest/dg/t_updating-inserting-using-staging-tables-.html||target="blank"]].
17 +* Job dashboard (See Deploy -> Runtime Dashboard -> [Select Runtime] - > [Select Flow] -> Jobs ).
18 +* Refresh an AWS Materialized View a minute after the job completes successfully.
19 +
20 +==== 2. Keynotes & restrictions ====
21 +
22 +* Source data must be shared in Mendix using a 'Published OData service'. If applicable in the settings tab set the association configuration to 'As an associated object id'.
23 +* It is highly recommended to configure Amazon Redshift to access the S3 bucket using an IAM role (documentation).
24 +* Mendix application must be reachable by the eMagiz runtime.
25 +* Before use, please rename job 'system.message.job' and step 'system.message.step1' to match your system and integration name. Make sure to update support.job-launch-request accordingly.
26 +* When the SQL statement fails, the step and job execution will still be registered as being 'completed'. Please look for log entries like 'Exception while closing step execution resources in step' if data is missing in Amazon Redshift.
27 +* When using multiple pipeline flows on one runtime, an h2 'already in use' exception might occur. To prevent these errors please keep the name of the h2 database different for each flow on one runtime by renaming the 'dp.h2.message.database' deploy property in the "support.h2-database" component.
28 +* When you want to import the flow multiple times for one system, you should change the names of the following properties by replacing 'message' with the name of the message type of each integration:
29 +** dp.odata.message.cron (component: receive.cron)
30 +** dp.odata.message.url (component: system.message.job -> system.message.step1 -> Item reader)
31 +** dp.h2.message.database (component: support.h2-database)
32 +** dp.jdbc.message.tablename (component: transform.job-launch-request - > Expression)
33 +** dp.jdbc.message.primarykey (component: transform.job-launch request -> Expression)
34 +
35 +==== 3. License Information ====
36 +
17 17  Part of the standard license agreement as agreed upon when using the store for the first time.
18 18  
19 -**4. Relevant eMagiz Academy Microlearnings**
20 -[[https:~~/~~/emagiz.github.io/docs/microlearning/novice-soap-webservice-connectivity-authorization-calling-a-soap-webservice>>url:https://emagiz.github.io/docs/microlearning/novice-soap-webservice-connectivity-authorization-calling-a-soap-webservice]]
39 +==== 4. Relevant eMagiz Academy Microlearnings ====
21 21  
41 +* [[eMagiz Data pipelines>>doc:Main.eMagiz Academy.Microlearnings.Legacy Functionality.intermediate-data-pipelines-understanding-data-pipelines.WebHome||target="blank"]]
22 22  
23 -Would you please consult the privacy policy of eMagiz at the following link: [[https:~~/~~/www.emagiz.com/privacy-policy/?>>url:https://www.emagiz.com/privacy-policy/?]])))
43 +//Would you please consult the privacy policy of eMagiz at the following [[link>>https://www.emagiz.com/privacy-policy/||target="blank"]]//
44 +)))
24 24  
25 25  
26 26  
27 -((((% border="2" cellpadding="10" cellspacing="10" style="width:292px" %)
28 -|=(% style="width: 45px;" %)#|=(% style="width: 241px;" %)Option
29 -|(% style="width:45px" %) a|(% style="width:241px" %) Basic Authentication
30 -|(% style="width:45px" %) b|(% style="width:241px" %) WS Addressing - Call back
31 -|(% style="width:45px" %) c|(% style="width:241px" %) Client certificate
32 -|(% style="width:45px" %) d|(% style="width:241px" %) Mapped headers
33 -|(% style="width:45px" %) e|(% style="width:241px" %) Token based - static
34 -|(% style="width:45px" %) f|(% style="width:241px" %) Token based - dynamic
35 -|(% style="width:45px" %) g|(% style="width:241px" %) SSL
36 -|(% style="width:45px" %) h|(% style="width:241px" %) No authentication
37 -|(% style="width:45px" %) i|(% style="width:241px" %) Dynamic URL
48 +(((
49 +
38 38  )))
39 39  {{/container}}
40 40  {{/container}}