Skip to Content

Changes for page SMB Connectivity

Last modified by Danniar Firdausy on 2024/09/18 20:40
From version 35.12
edited by Danniar Firdausy
on 2024/09/18 20:38
Change comment: There is no comment for this version
To version 8.1
edited by Bouke Reitsma
on 2024/05/06 10:29
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.dfirdausy
1 +XWiki.BoukeReitsma
Content
... ... @@ -1,6 +1,5 @@
1 -{{container}}
2 -{{container layoutStyle="columns"}}(((
3 -In many scenarios, accessing or managing files directly through a local directory isn't feasible, or external requirements might necessitate file placement in different locations. Typically, file transfers are handled via FTP or SFTP protocols, but another option available is Server Message Block (SMB) connectivity. In this microlearning, we will explore how SMB connectivity works within eMagiz, examining how it facilitates file transfers between local and remote systems.
1 +{{container}}{{container layoutStyle="columns"}}(((
2 +In some cases, there is no communication possible with a local file directory. Or there is a requirement from the external party to place the files somewhere else. Mainly the somewhere else is either an FTP or an SFTP. However, there is a third communication protocol that can be used for file transfer. In this microlearning we will take a look at Server Message Block (SMB) connectivity within eMagiz.
4 4  
5 5  Should you have any questions, please get in touch with [[academy@emagiz.com>>mailto:academy@emagiz.com]].
6 6  
... ... @@ -12,8 +12,14 @@
12 12  == 2. Key concepts ==
13 13  
14 14  This microlearning is about SMB (connectivity).
15 -* By SMB connectivity, we mean: Making sure that eMagiz can communicate with the SMB protocol to exchange file-based data.
16 16  
15 +By SMB connectivity, we mean: Making sure that eMagiz can communicate with the SMB protocol to exchange file-based data.
16 +
17 +* SMB (Server Message Block) is a network protocol for transmitting files between computers over Transmission Control Protocol/Internet Protocol (TCP/IP) connections.
18 +* In an SMB transaction, the end user's computer is typically called the localhost. The second computer involved in SMB is a remote host, which is usually a server.
19 +* Servers must be set up to run SMB services, and the client must have SMB software installed to access these services.
20 +* eMagiz provides several components to set up the SMB connection
21 +
17 17  == 3. SMB Connectivity ==
18 18  
19 19  In some cases, there is no communication possible with a local file directory. Or there is a requirement from the external party to place the files somewhere else. Mainly the somewhere else is either an FTP or an SFTP. However, there is a third communication protocol that can be used for file transfer. In this microlearning we will take a look at Server Message Block (SMB) connectivity within eMagiz.
... ... @@ -23,104 +23,104 @@
23 23  * Servers must be set up to run SMB services, and the client must have SMB software installed to access these services.
24 24  * eMagiz provides several components to set up the SMB connection
25 25  
31 +-----------------------------------------------------------------------------------------------------------------------------------
32 +
26 26  So, where the file pickup component searches on the local host for the correct directory, the idea of the SMB setup is that the directory with whom to communicate exists on a remote host. The local host and the remote host can talk to each other via the SMB protocol. That is, in a nutshell, the difference between setting up an SMB connection and communicating with local directories.
27 27  
28 -{{warning}}Note that the auto create directory setting on all SMB components only works for a "one-level" directory (i.e. /directory) and not for a "multi-level" directory (i.e. /directory/sub). To ensure this "multi-level" directory is created by eMagiz you need to take a look at this [[microlearning>>doc:Main.eMagiz Academy.Microlearnings.Advanced Level.File based connectivity.advanced-file-connectivity-dynamically-manipulate-smb-settings.WebHome||target="blank"]]{{/warning}}
35 +eMagiz provides only one support objects to connect to an SMB:
29 29  
30 -eMagiz provides only one support object to connect to an SMB:
37 +* Default SMB session factory
31 31  
32 -* SMB session factory
39 +[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--ftp-support-objects.png]]
33 33  
34 -[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-smb-connectivity-smb-support-object.png]]
41 +=== 3.1 Best practices for using FTP ===
35 35  
36 -=== 3.1 Configuration in eMagiz ===
43 +==== 3.1.1 Which session factory do I need? ====
37 37  
38 -To configure the connection between eMagiz and the SMB, we need two (or three) components. This depends on whether we want to read, write or do both while communicating with the SMB. If we're going to do both, we need both of the functional components. In all other scenarios, you only need one of the functional components. The functional components that you might need are called:
45 +When occasionally retrieving or writing files to FTP, use the _Default FTP session factory_. A new connection will be set up when you use the _Default FTP session factory_ for every message/file. On some FTP servers, this might take up to seconds for each login. This fact might impact the performance of your flow.
39 39  
40 -* SMB inbound Channel adapter
41 -* SMB outbound Channel adapter
47 +When you need better performance (say writing multiple files in a minute or retrieving files every minute), use the _Default FTP **caching** session factory_. The _Default FTP **caching** session factory_ shares FTP sessions for messages in your flow.
42 42  
43 -On top of that, you need the following support object.
49 +==== 3.1.2 Do I have to set up a retry mechanism? ====
44 44  
45 -* SMB session factory
51 +When writing files to FTP, it is advisable to set a retry mechanism. Find the advanced tab on your _FTP outbound channel adapter_. Under _Request handler advice chain_ add a _Retry Advice_.
46 46  
47 -In this microlearning, we will use the SMB inbound channel adapter combined with the SMB session factory to illustrate the configuration. Please note that any other combination works similarly. To configure the connection with the SMB, we first navigate to the Create phase of eMagiz. Here we open the flow we want to edit. In this example, that will be an entry flow as we want to retrieve (read) data from the SMB.
53 +==== 3.1.3 I get a _Failed to list files/items_ or a _Failed to obtain pooled item_. What can I do? ====
48 48  
49 -When you have opened the flow, you need to enter "Start Editing" mode. After you have done so, we first add the SMB session factory support object to our flow.
55 +Incorrect connection settings can cause this. Check whether the correct properties have been set and whether the runtime has retrieved the latest properties.
50 50  
51 -[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-smb-connectivity-smb-session-factory-empty.png]]
57 +==== 3.1.4 I get a _Failed to list files / items_, a _Pipe closed_ or an _input stream is closed_. What can I do? ====
52 52  
53 -Here, we need to define the following:
59 +A closed FTP session can cause these notifications due to inactivity. For the _Default FTP **caching** session factory_, always set the _Server alive interval_ property to 30000 (ms).
54 54  
55 -* Host
56 -* Port
57 -* Username
58 -* Password
59 -* Share and directory
61 +==== 3.1.6 I get a _Failed to write to_. What can I do? ====
60 60  
61 -{{info}}
62 -The value for the share and directory field is the main folder for your file share. If the main folder is the exchange folder, then within the SMB inbound channel adapter, a / is needed as remote directory.
63 -{{/info}}
63 +Maybe the target folder does not exist, and the _Auto-create directory_ is off. It is also possible that the connection settings are incorrect. Check whether the correct properties have been set and whether the runtime has retrieved the latest properties.
64 64  
65 -[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-smb-connectivity-smb-session-factory-filled-in.png]]
65 +==== 3.1.7 I often get 'Disconnecting from sftp.example.com port 22' and then a 'Caught an exception, leaving main loop due to Socket closed' warnings. ====
66 66  
67 -In the advanced tab there are only two settings to change. The minimum and maximum SMB protocol version. By default, these are set to 2.02 and 3.1.1 respectively to allow a wide range of save protocol version. However, if you want to only allow the best performing version, the minimum 3.0.2 is advised. If you aim for the best secured connection version 3.1.1 has the most functionality in this area. The best practise on this is set the minimum version as high as possible to mimize risks.
67 +Too many connections can cause this to the same server at the same time. If you have many flows connecting to the same FTP server, ensure that the crons do not trigger simultaneously.
68 68  
69 -Now that we have configured the support object,we can add the SMB inbound channel adapter to the flow.
69 +==== 3.1.8 I get a 'Failed to create SFTPClient' ====
70 70  
71 -[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-smb-connectivity-smb-inbound-channel-adapter-empty.png]]
71 +SFTP, FTP, and FTPS are three different remote file transfer protocols with their _session factory_ components in the eMagiz flow designer. Make sure you use the one that corresponds to the server's protocol.
72 72  
73 -In this component, we need to define the remote directory where the original files are located. We need to specify the local directory used to store the files before they are processed temporarily. Furthermore, we need to link the support object we have just created and define a poller. For that last one, the best practice is the use of a cron trigger.
73 +==== 3.1.9 One of my remote files is not picked up from the SFTP server (without any warning) ====
74 74  
75 -[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-smb-connectivity-smb-inbound-channel-adapter-filled-in.png]]
75 +The default local filter is _AcceptOnceFileListFilter_. This can be replaced by your file list filter on the _FTP inbound channel adapter's_.
76 +If every day you retrieve a file with the same filename and the _Delete remote files_ property is set, it is advised to use a different local filename. You can add the current timestamp, for example.
76 76  
77 -Apart from these configurations, you could also add a filter, just as we learned in the [[Crash Course Messaging on file-based connectivity>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Messaging.crashcourse-messaging-pick-up-files||target="blank"]]. Finishing things off, we should consider the delete behavior we will use. Based on the filter and the business requirements, whether to delete and when to delete could differ. So please consider all your options in that regard. In case there are multiple readers from the folder and/or there are legal requirements to store source files for a certain time period, you should not use the delete remote files option. However, not deleting files read files can cause overloading the machine if no alternative delete policies is in place.
78 +==== 3.1.10 Which mode is needed ====
78 78  
79 -=== 3.2 SMB Gateway ===
80 +When running in the current-generation architecture (G3) the FTP always needs to run **passive** mode.
80 80  
81 -The SMB Gateway is an additional component that can be used to interact with your SMB share. It allows the use of a set of pre-configured commands to perform actions such as deleting files. Compared to the outbound channel adapter replies are send back to eMagiz. These reply messages depend on the selected command. Every command option is explained in detail within the helptext. An example configuration to send files is shown below.
82 +=== 3.2 Configuration in eMagiz ===
82 82  
83 -[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-smb-connectivity-smb-outbound-gateway-filled-in.png]]
84 +To configure the connection between eMagiz and the FTP, we need two (or three) components. This depends on whether we want to read, write or do both while communicating with the FTP. If we're going to do both, we need both of the functional components. In all other scenarios, you only need one of the functional components. The functional components that you might need are called:
84 84  
85 -{{info}}
86 -Please note that for some commands an SpEL expression is required to respresent a path. This setting can be found in the advanced settings. An example for the mget command could be **myfilesdirectory/** to get only files from this directory.
87 -{{/info}}
86 +* FTP Inbound Channel adapter
87 +* FTP Outbound Channel adapter
88 88  
89 -All available options are:
89 +On top of that, you need one of the following support objects. If you are unsure of which to use, please check out the information above in this microlearning.
90 90  
91 -* List files
92 -* List file names
93 -* Retrieve file
94 -* Retrieve multiple files
95 -* Remove file(s)
96 -* Move/rename file
97 -* Send file
98 -* Send mulitple files
91 +* Default FTP caching session factory
92 +* Default FTP session factory
99 99  
100 -This concludes our microlearning, in which we learned how to set up the connectivity with an SMB.
94 +In this microlearning, we will use the FTP Inbound channel adapter combined with the Default FTP session factory to illustrate the configuration. Please note that any other combination works similarly. To configure the connection with the FTP, we first navigate to the Create phase of eMagiz. Here we open the flow we want to edit. In this example, that will be an entry flow as we want to retrieve (read) data from the FTP.
101 101  
96 +When you have opened the flow, you need to enter "Start Editing" mode. After you have done so, we first add the support object to our flow. In this case, we will use the Default FTP session factory.
97 +
98 +[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--default-ftp-session-factory-empty.png]]
99 +
100 +Here, we need to define at least the host and the port. Note that the username and password are optional for an FTP, which makes it less secure. Although in practice, we generally see that the FTP is secured with the help of a username-password combination. As a best practice, we use property references to fill in the various elements.
101 +
102 +[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--default-ftp-session-factory-filled-in.png]]
103 +
104 +The settings on the Advanced tab don't have to be changed in most cases. If your specific setup does require a change to these settings, please consult the help texts on the component level for advice.
105 +
106 +Now that we have configured the support object adding the FTP inbound channel adapter to the flow has become time.
107 +
108 +[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--ftp-inbound-channel-adapter-empty.png]]
109 +
110 +In this component, we need to define the remote directory where the original files are located. We need to specify the local directory used to store the files before they are processed temporarily. Furthermore, we need to link the support object we have just created and define a poller. For that last one, the best practice is the use of a cron trigger.
111 +
112 +[[image:Main.Images.Microlearning.WebHome@intermediate-file-based-connectivity-ftp-connectivity--ftp-inbound-channel-adapter-filled-in.png]]
113 +
114 +Apart from these configurations, you could also add a filter, just as we learned in the Crash Course Messaging on file-based connectivity. Finishing things off, we should consider the delete behavior we will use. Based on the filter and the business requirements, whether to delete and when to delete could differ. So please consider all your options in that regard. This concludes our microlearning, in which we learned how to set up the connectivity with an FTP.
115 +
102 102  == 4. Key takeaways ==
103 103  
104 -* Server Message Block (SMB) is a network protocol used for sharing files between computers over TCP/IP connections. It allows a client to access files on a remote server as if they were on the local machine.
105 -* In an SMB transaction, the end user's computer is typically called the localhost. The second computer involved in SMB is a remote host, which is usually a server.
106 -* Servers must be set up to run SMB services, and the client must have SMB software installed to access these services.
107 -* To use SMB with eMagiz, you'll need the SMB session factory and either an SMB inbound or outbound channel adapter, depending on whether you're reading from or writing to the SMB share. The SMB Gateway component offers pre-configured commands for managing files, including options to list, retrieve, move, or delete files.
118 +* FTP (File Transfer Protocol) is a network protocol for transmitting files between computers over Transmission Control Protocol/Internet Protocol (TCP/IP) connections.
119 +* In an FTP transaction, the end user's computer is typically called the localhost. The second computer involved in FTP is a remote host, which is usually a server.
120 +* Servers must be set up to run FTP services, and the client must have FTP software installed to access these services.
121 +* eMagiz provides several components to set up the FTP connection
108 108  
109 109  == 5. Suggested Additional Readings ==
110 110  
111 111  If you are interested in this topic and want more information, please read the release notes provided by eMagiz. Furthermore, check out these links:
112 112  
113 -* [[eMagiz Store (Menu)>>doc:Main.eMagiz Store.WebHome||target="blank"]]
114 -** [[Accelerators (Navigation)>>doc:Main.eMagiz Store.Accelerators.WebHome||target="blank"]]
115 -*** [[SMB Inbound Connectivity (Explanation)>>doc:Main.eMagiz Store.Accelerators.SFTP Inbound Connectivity.WebHome||target="blank"]]
116 -*** [[SMB Outbound Connectivity (Explanation)>>doc:Main.eMagiz Store.Accelerators.SFTP Outbound Connectivity.WebHome||target="blank"]]
117 -*** [[SMB Gateway Connectivity (Explanation)>>doc:Main.eMagiz Store.Accelerators.SFTP Gateway Connectivity.WebHome||target="blank"]]
118 -* [[Crash Course (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.WebHome||target="blank"]]
119 -** [[Crash Course Messaging (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Messaging.WebHome||target="blank"]]
120 -*** [[Pick up Files (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Messaging.crashcourse-messaging-pick-up-files||target="blank"]]
121 -* [[SMB (Search Results)>>url:https://docs.emagiz.com/bin/view/Main/Search?sort=score&sortOrder=desc&highlight=true&facet=true&r=1&f_space_facet=0%2FMain.&l_space_facet=10&f_type=DOCUMENT&f_locale=en&f_locale=&f_locale=en&text=%22SMB%22||target="blank"]]
122 -* [[SMB Overview (External)>>https://learn.microsoft.com/en-us/windows-server/storage/file-server/file-server-smb-overview||target="blank"]]
123 -* [[SMB Support (External)>>https://docs.spring.io/spring-integration/reference/smb.html||target="blank"]]
127 +* [[FTP Inbound>>https://docs.spring.io/spring-integration/docs/2.1.x/reference/html/ftp.html#ftp-inbound||target="blank"]]
128 +* [[FTP Session Factory>>https://docs.spring.io/spring-integration/docs/4.3.x/reference/html/ftp.html#ftp-session-factory||target="blank"]]
129 +* [[FTP Explained>>https://searchnetworking.techtarget.com/definition/File-Transfer-Protocol-FTP||target="blank"]]
124 124  
125 -)))((({{toc/}}))){{/container}}
126 -{{/container}}
131 +)))((({{toc/}}))){{/container}}{{/container}}