Wiki source code of FTP Connectivity

Last modified by Bouke Reitsma on 2024/05/13 15:21
Show last authors
1 {{container}}{{container layoutStyle="columns"}}(((
2 In some cases, there is no communication possible with a local file directory. Or there is a requirement from the external party to place the files somewhere else. Mainly the somewhere else is either an FTP or an SFTP. In this microlearning, we will look at the FTP and learn how to set up the connection between eMagiz and the FTP.
3
4 Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
5
6 == 1. Prerequisites ==
7
8 * Basic knowledge of the eMagiz platform
9 * Novice course on File-based connectivity
10
11 == 2. Key concepts ==
12
13 This microlearning is about FTP (connectivity).
14
15 By FTP connectivity, we mean: Making sure that eMagiz can communicate with the FTP to exchange file-based data.
16
17 * FTP (File Transfer Protocol) is a network protocol for transmitting files between computers over Transmission Control Protocol/Internet Protocol (TCP/IP) connections.
18 * In an FTP transaction, the end user's computer is typically called the localhost. The second computer involved in FTP is a remote host, which is usually a server.
19 * Servers must be set up to run FTP services, and the client must have FTP software installed to access these services.
20 * eMagiz provides several components to set up the FTP connection
21
22 == 3. FTP Connectivity ==
23
24 In some cases, there is no communication possible with a local file directory. Or there is a requirement from the external party to place the files somewhere else. Mainly the somewhere else is either an FTP or an SFTP. This microlearning will look at the FTP and learn how to set up the connection between eMagiz and the FTP.
25
26 * FTP (File Transfer Protocol) is a network protocol for transmitting files between computers over Transmission Control Protocol/Internet Protocol (TCP/IP) connections.
27 * In an FTP transaction, the end user's computer is typically called the localhost. The second computer involved in FTP is a remote host, which is usually a server.
28 * Servers must be set up to run FTP services, and the client must have FTP software installed to access these services.
29 * eMagiz provides several components to set up the FTP connection
30
31 So, where the file pickup component searches on the local host for the correct directory, the idea of the FTP setup is that the directory with whom to communicate exists on a remote host. The local host and the remote host can talk to each other via the FTP protocol. That is, in a nutshell, the difference between setting up an FTP connection and communicating with local directories.
32
33 eMagiz provides two support objects to connect to an FTP:
34
35 * Default FTP caching session factory
36 * Default FTP session factory
37
38 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--ftp-support-objects.png]]
39
40 The main difference between the two is the caching part. The caching session factory comes with the option to re-use the existing session for multiple requests. In other words, this way, you do not have to authenticate yourself every time you want to communicate with the FTP. This behavior is beneficial if a high frequency of communication is needed between eMagiz and the FTP to handle the incoming and outgoing data packages.
41
42 If you only initiate communication infrequently, for example, once per day, the default session factory becomes more interesting as this closes the session after each transmission. This behavior is beneficial in cases of low-frequency communication because otherwise, the session would remain active for an extensive period.
43
44 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--ftp-caching-help-text.png]]
45
46 === 3.1 Best practices for using FTP ===
47
48 ==== 3.1.1 Which session factory do I need? ====
49
50 When occasionally retrieving or writing files to FTP, use the _Default FTP session factory_. A new connection will be set up when you use the _Default FTP session factory_ for every message/file. On some FTP servers, this might take up to seconds for each login. This fact might impact the performance of your flow.
51
52 When you need better performance (say writing multiple files in a minute or retrieving files every minute), use the _Default FTP **caching** session factory_. The _Default FTP **caching** session factory_ shares FTP sessions for messages in your flow.
53
54 ==== 3.1.2 Do I have to set up a retry mechanism? ====
55
56 When writing files to FTP, it is advisable to set a retry mechanism. Find the advanced tab on your _FTP outbound channel adapter_. Under _Request handler advice chain_ add a _Retry Advice_.
57
58 ==== 3.1.3 I get a _Failed to list files/items_ or a _Failed to obtain pooled item_. What can I do? ====
59
60 Incorrect connection settings can cause this. Check whether the correct properties have been set and whether the runtime has retrieved the latest properties.
61
62 ==== 3.1.4 I get a _Failed to list files / items_, a _Pipe closed_ or an _input stream is closed_. What can I do? ====
63
64 A closed FTP session can cause these notifications due to inactivity. For the _Default FTP **caching** session factory_, always set the _Server alive interval_ property to 30000 (ms).
65
66 ==== 3.1.6 I get a _Failed to write to_. What can I do? ====
67
68 Maybe the target folder does not exist, and the _Auto-create directory_ is off. It is also possible that the connection settings are incorrect. Check whether the correct properties have been set and whether the runtime has retrieved the latest properties.
69
70 ==== 3.1.7 I often get 'Disconnecting from sftp.example.com port 22' and then a 'Caught an exception, leaving main loop due to Socket closed' warnings. ====
71
72 Too many connections can cause this to the same server at the same time. If you have many flows connecting to the same FTP server, ensure that the crons do not trigger simultaneously.
73
74 ==== 3.1.8 I get a 'Failed to create SFTPClient' ====
75
76 SFTP, FTP, and FTPS are three different remote file transfer protocols with their _session factory_ components in the eMagiz flow designer. Make sure you use the one that corresponds to the server's protocol.
77
78 ==== 3.1.9 One of my remote files is not picked up from the SFTP server (without any warning) ====
79
80 The default local filter is _AcceptOnceFileListFilter_. This can be replaced by your file list filter on the _FTP inbound channel adapter's_.
81 If every day you retrieve a file with the same filename and the _Delete remote files_ property is set, it is advised to use a different local filename. You can add the current timestamp, for example.
82
83 ==== 3.1.10 Which mode is needed ====
84
85 When running in the current-generation architecture (G3) the FTP always needs to run **passive** mode.
86
87 === 3.2 Configuration in eMagiz ===
88
89 To configure the connection between eMagiz and the FTP, we need two (or three) components. This depends on whether we want to read, write or do both while communicating with the FTP. If we're going to do both, we need both of the functional components. In all other scenarios, you only need one of the functional components. The functional components that you might need are called:
90
91 * FTP Inbound Channel adapter
92 * FTP Outbound Channel adapter
93
94 On top of that, you need one of the following support objects. If you are unsure of which to use, please check out the information above in this microlearning.
95
96 * Default FTP caching session factory
97 * Default FTP session factory
98
99 In this microlearning, we will use the FTP Inbound channel adapter combined with the Default FTP session factory to illustrate the configuration. Please note that any other combination works similarly. To configure the connection with the FTP, we first navigate to the Create phase of eMagiz. Here we open the flow we want to edit. In this example, that will be an entry flow as we want to retrieve (read) data from the FTP.
100
101 When you have opened the flow, you need to enter "Start Editing" mode. After you have done so, we first add the support object to our flow. In this case, we will use the Default FTP session factory.
102
103 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--default-ftp-session-factory-empty.png]]
104
105 Here, we need to define at least the host and the port. Note that the username and password are optional for an FTP, which makes it less secure. Although in practice, we generally see that the FTP is secured with the help of a username-password combination. As a best practice, we use property references to fill in the various elements.
106
107 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--default-ftp-session-factory-filled-in.png]]
108
109 The settings on the Advanced tab don't have to be changed in most cases. If your specific setup does require a change to these settings, please consult the help texts on the component level for advice.
110
111 Now that we have configured the support object adding the FTP inbound channel adapter to the flow has become time.
112
113 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--ftp-inbound-channel-adapter-empty.png]]
114
115 In this component, we need to define the remote directory where the original files are located. We need to specify the local directory used to store the files before they are processed temporarily. Furthermore, we need to link the support object we have just created and define a poller. For that last one, the best practice is the use of a cron trigger.
116
117 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--ftp-inbound-channel-adapter-filled-in.png]]
118
119 Apart from these configurations, you could also add a filter, just as we learned in the Crash Course Messaging on file-based connectivity. Finishing things off, we should consider the delete behavior we will use. Based on the filter and the business requirements, whether to delete and when to delete could differ. So please consider all your options in that regard. This concludes our microlearning, in which we learned how to set up the connectivity with an FTP.
120
121 === 3.2 FTP Gateway ===
122
123 The FTP Gateway is an additional component that can be used to interact with your FTP share. It enables the use of a set of pre-configured commands to for example delete files. The reply messages depend on the selected command. Every command option is explained within the helptext. An example configuration to send files is shown below.
124
125 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity-ftp-outbound-gateway-filled-in.png]]
126
127 {{info}}
128 Please note that for some commands an SpEL expression is required to respresent a path. This setting can be found in the advanced settings. An example for the mget command could be **myfilesdirectory/** to get only files from this directory.
129 {{/info}}
130
131 All available options are:
132
133 * List files
134 * List file names
135 * Retrieve file
136 * Retrieve multiple files
137 * Remove file(s)
138 * Move/rename file
139 * Send file
140 * Send mulitple files
141
142 This concludes our microlearning, in which we learned how to set up the connectivity with an FTP.
143
144 == 4. Key takeaways ==
145
146 * FTP (File Transfer Protocol) is a network protocol for transmitting files between computers over Transmission Control Protocol/Internet Protocol (TCP/IP) connections.
147 * In an FTP transaction, the end user's computer is typically called the localhost. The second computer involved in FTP is a remote host, which is usually a server.
148 * Servers must be set up to run FTP services, and the client must have FTP software installed to access these services.
149 * eMagiz provides several components to set up the FTP connection
150
151 == 5. Suggested Additional Readings ==
152
153 If you are interested in this topic and want more information, please read the release notes provided by eMagiz. Furthermore, check out these links:
154
155 * [[FTP Inbound>>https://docs.spring.io/spring-integration/docs/2.1.x/reference/html/ftp.html#ftp-inbound||target="blank"]]
156 * [[FTP Session Factory>>https://docs.spring.io/spring-integration/docs/4.3.x/reference/html/ftp.html#ftp-session-factory||target="blank"]]
157 * [[FTP Explained>>https://searchnetworking.techtarget.com/definition/File-Transfer-Protocol-FTP||target="blank"]]
158
159 )))((({{toc/}}))){{/container}}{{/container}}