Wiki source code of SFTP Connectivity

Last modified by Bouke Reitsma on 2024/05/13 15:18
Show last authors
1 {{container}}{{container layoutStyle="columns"}}(((
2 In some cases, there is no communication possible with a local file directory. Or there is a requirement from the external party to place the files somewhere else. Mainly the somewhere else is either an FTP or an SFTP. This microlearning will look at the SFTP and learn how to set up the connection between eMagiz and the SFTP.
3
4 Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
5
6 == 1. Prerequisites ==
7
8 * Basic knowledge of the eMagiz platform
9 * Novice course on File-based connectivity
10
11 == 2. Key concepts ==
12
13 This microlearning is about SFTP (connectivity).
14
15 By SFTP connectivity, we mean: Making sure that eMagiz can communicate with the SFTP to exchange file-based data.
16
17 * Secure File Transfer Protocol (SFTP) is a file protocol for transferring large files over the web. It builds on the File Transfer Protocol (FTP) and includes Secure Shell (SSH) security components.
18 * In an SFTP transaction, the end user's computer is typically called the localhost. The second computer involved in SFTP is a remote host, which is usually a server.
19 * Servers must be set up to run SFTP services, and the client must have SFTP software installed to access these services.
20 * eMagiz provides several components to set up the SFTP connection
21
22 == 3. SFTP Connectivity ==
23
24 In some cases, there is no communication possible with a local file directory. Or there is a requirement from the external party to place the files somewhere else. Mainly the somewhere else is either an FTP or an SFTP. This microlearning will look at the FTP and learn how to set up the connection between eMagiz and the SFTP.
25
26 * Secure File Transfer Protocol (SFTP) is a file protocol for transferring large files over the web. It builds on the File Transfer Protocol (FTP) and includes Secure Shell (SSH) security components.
27 * In an SFTP transaction, the end user's computer is typically called the localhost. The second computer involved in SFTP is a remote host, which is usually a server.
28 * Servers must be set up to run SFTP services, and the client must have SFTP software installed to access these services.
29 * eMagiz provides several components to set up the SFTP connection
30
31 So, where the file pickup component searches on the local host for the correct directory, the idea of the SFTP setup is that the directory with whom to communicate exists on a remote host. The local host and the remote host can talk to each other via the SFTP protocol. This is quite similar to FTP, with the critical difference that the S in SFTP stands for Secure. This functionality avoids password sniffing and exposing sensitive information in plain text. Since the client needs to be authenticated by the server, SFTP also protects against man-in-the-middle attacks. That is, in a nutshell, the difference between setting up an SFTP connection, setting up an FTP connection, and communicating with local directories.
32
33 eMagiz provides two support objects to connect to an SFTP:
34
35 * Default SFTP caching session factory
36 * Default SFTP session factory
37
38 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-sftp-connectivity--sftp-support-objects.png]]
39
40 The main difference between the two is the caching part. The caching session factory comes with the option to re-use the existing session for multiple requests. In other words, this way, you do not have to authenticate yourself every time you want to communicate with the SFTP. This behavior is beneficial if a high frequency of communication is needed between eMagiz and the SFTP to handle the incoming and outgoing data packages.
41
42 If you only initiate communication infrequently, for example, once per day, the default session factory becomes more interesting as this closes the session after each transmission. This behavior is beneficial in cases of low-frequency communication because otherwise, the session would remain active for an extensive period.
43
44 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-sftp-connectivity--sftp-caching-help-text.png]]
45
46 === 3.1 Best practices for using SFTP ===
47
48 ==== 3.1.1 Which session factory do I need? ====
49
50 When occasionally retrieving or writing files to SFTP, use the _Default SFTP session factory_. A new connection will be set up when you use the _Default SFTP session factory_ for every message/file. On some SFTP servers, this might take up to seconds for each login. This fact might impact the performance of your flow.
51
52 When you need better performance (say writing multiple files in a minute or retrieving files every minute), use the _Default SFTP **caching** session factory_. The _Default SFTP **caching** session factory_ shares FTP sessions for messages in your flow.
53
54 ==== 3.1.2 Do I have to set up a retry mechanism? ====
55
56 When writing files to SFTP, it is advisable to set a retry mechanism. Find the advanced tab on your _SFTP outbound channel adapter_. Under _Request handler advice chain_ add a _Retry Advice_.
57
58 ==== 3.1.3 I get a _Failed to list files/items_ or a _Failed to obtain pooled item_. What can I do? ====
59
60 Incorrect connection settings can cause this. Check whether the correct properties have been set and whether the runtime has retrieved the latest properties.
61
62 ==== 3.1.4 I get a _Failed to list files / items_, a _Pipe closed_ or an _input stream is closed_. What can I do? ====
63
64 A closed SFTP session can cause these notifications due to inactivity. For the _Default SFTP **caching** session factory_, always set the _Server alive interval_ property to 30000 (ms).
65
66 ==== 3.1.6 I get a _Failed to write to_. What can I do? ====
67
68 Maybe the target folder does not exist, and the _Auto-create directory_ is off. It is also possible that the connection settings are incorrect. Check whether the correct properties have been set and whether the runtime has retrieved the latest properties.
69
70 ==== 3.1.7 I often get 'Disconnecting from sftp.example.com port 22' and then a 'Caught an exception, leaving main loop due to Socket closed' warnings. ====
71
72 Too many connections can cause this to the same server at the same time. If you have many flows connecting to the same SFTP server, ensure that the crons do not trigger simultaneously.
73
74 ==== 3.1.8 I get a 'Failed to create SFTPClient' ====
75
76 SFTP, FTP, and FTPS are three different remote file transfer protocols with their _session factory_ components in the eMagiz flow designer. Make sure you use the one that corresponds to the server's protocol.
77
78 ==== 3.1.9 One of my remote files is not picked up from the SFTP server (without any warning) ====
79
80 The default local filter is _AcceptOnceFileListFilter_. This can be replaced by your file list filter on the _SFTP inbound channel adapter's_.
81 If every day you retrieve a file with the same filename and the _Delete remote files_ property is set, it is advised to use a different local filename. You can add the current timestamp, for example.
82
83 === 3.2 Configuration in eMagiz ===
84
85 To configure the connection between eMagiz and the SFTP, we need two (or three) components. This depends on whether we want to read, write or do both while communicating with the FTP. If we're going to do both, we need both of the functional components. In all other scenarios, you only need one of the functional components. The functional components that you might need are called:
86
87 * SFTP Inbound Channel adapter
88 * SFTP Outbound Channel adapter
89
90 On top of that, you need one of the following support objects. If you are unsure of which to use, please check out the information above in this microlearning.
91
92 * Default SFTP caching session factory
93 * Default SFTP session factory
94
95 This microlearning will use the SFTP Outbound channel adapter combined with the Default SFTP caching session factory to illustrate the configuration. Please note that any other combination works similarly. To configure the connection with the SFTP, we first navigate to the Create phase of eMagiz. Here we open the flow we want to edit. In this example, that will be an exit flow as we want to write data to the SFTP.
96
97 When you have opened the flow, you need to enter "Start Editing" mode. After you have done so, we first add the support object to our flow. In this case, we will use the Default SFTP caching session factory.
98
99 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-sftp-connectivity--default-sftp-session-factory-empty.png]]
100
101 Here, we need to define the following:
102
103 * Host
104 * Port
105 * Username
106 * Password
107 * Known hosts (and Allow unknown keys the first time)
108
109 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-sftp-connectivity--default-sftp-session-factory-filled-in.png]]
110
111 Note that the setup we have chosen in this example is the setup you should run your flow the first time. Afterward, ensure to uncheck the Allow unknown keys checkbox to safeguard yourself against so-called man-in-the-middle attacks. For more detailed information, please check out the help text provided on that setting.
112
113 There is also the option to use the private key option in combination with a passphrase. More on that in our next microlearning. The settings on the Advanced tab don't have to be changed in most cases. If your specific setup does require a change to these settings, please consult the help texts on the component level for advice.
114
115 Now that we have configured the support object adding the SFTP outbound channel adapter to the flow has become time.
116
117 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-sftp-connectivity--sftp-outbound-channel-adapter-empty.png]]
118
119 In this component, we need to define the remote directory where the files need to be placed. Furthermore, we need to link the support object we have just created and decide whether to auto-create the directory. I always leave this on false as it gives me control over creating the correct file directory the moment I want to make it. Others might find it a valuable option to create it automatically.
120
121 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-sftp-connectivity--sftp-outbound-channel-adapter-filled-in.png]]
122
123 Apart from these configurations on the basic tab, we also need to fill in two configuration options on the Advanced level. The first thing to determine is the Mode. Here you can choose which of the various Modes of writing data to the SFTP is relevant. More on that function can be learned in this [microlearning](crashcourse-messaging-drop-files#33-what-to-do-when-a-file-already-exists). If you use Replace (the default), you do not have to fill in anything. The second part is to add a request handler advice chain (this is a general best practice when communicating with the outside world via eMagiz).
124
125 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-sftp-connectivity--sftp-outbound-channel-adapter-filled-in-advanced.png]]
126
127 === 3.3 SFTP Gateway ===
128
129 The SFTP Gateway is an additional component that can be used to interact with your SFTP share. It enables the use of a set of pre-configured commands to for example delete files. The reply messages depend on the selected command. Every command option is explained within the helptext. An example configuration to send files is shown below.
130
131 [[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-sftp-connectivity-sftp-outbound-gateway-filled-in.png]]
132
133 {{info}}
134 Please note that for some commands an SpEL expression is required to respresent a path. This setting can be found in the advanced settings. An example for the mget command could be **myfilesdirectory/** to get only files from this directory.
135 {{/info}}
136
137 All available options are:
138
139 * List files
140 * List file names
141 * Retrieve file
142 * Retrieve multiple files
143 * Remove file(s)
144 * Move/rename file
145 * Send file
146 * Send mulitple files
147
148 This concludes our microlearning, in which we learned how to set up the connectivity with an SFTP.
149
150
151 == 4. Key takeaways ==
152
153 * Secure File Transfer Protocol (SFTP) is a file protocol for transferring large files over the web. It builds on the File Transfer Protocol (FTP) and includes Secure Shell (SSH) security components.
154 * In an SFTP transaction, the end user's computer is typically called the localhost. The second computer involved in SFTP is a remote host, which is usually a server.
155 * Servers must be set up to run SFTP services, and the client must have SFTP software installed to access these services.
156 * eMagiz provides several components to set up the SFTP connection
157
158 == 5. Suggested Additional Readings ==
159
160 If you are interested in this topic and want more information, please read the release notes provided by eMagiz. Furthermore, check out these links:
161
162 * [[SFTP Outbound>>https://docs.spring.io/spring-integration/docs/2.2.6.RELEASE/reference/html/sftp.html#sftp-outbound||target="blank"]]
163 * [[SFTP Session Factory>>https://docs.spring.io/spring-integration/docs/2.2.6.RELEASE/reference/html/sftp.html#sftp-session-factory||target="blank"]]
164 * [[SFTP Explained>>https://www.techopedia.com/definition/1879/secure-file-transfer-protocol-sftp||target="blank"]]
165
166 )))((({{toc/}}))){{/container}}{{/container}}