Wiki source code of Pick up Files

Last modified by Danniar Firdausy on 2024/08/21 16:48

version	line-number	content
17.1	1	{{container}}{{container layoutStyle="columns"}}(((
22.6	2	In this microlearning, we will explain the basics of retrieving files from a local directory using eMagiz. We will cover the essentials for setting up file retrieval from a directory that is directly accessible within the network where the eMagiz process is running. We will walk you through key concepts, considerations, and best practices to ensure a smooth file handling process.
1.1	3
22.6	4	If you have any questions along the way, feel free to reach out to us at [[academy@emagiz.com>>mailto:academy@emagiz.com]].
1.1	5
	6	== 1. Prerequisites ==
	7
	8	* Basic knowledge of the eMagiz platform
	9
	10	== 2. Key concepts ==
	11
	12	This microlearning centers around retrieving files from a local directory via eMagiz.
22.6	13	* With retrieving we mean: Checking with a certain interval if new files are ready to be pulled from the directory by emagiz
	14	* With local directory we mean: A directory that is directly accessible within the network where the eMagiz process is running
1.1	15
	16	== 3. Pick up Files ==
	17
	18	One of the easiest connectivity methods to implement within eMagiz is the retrieval of files from a local directory.
	19	With this method, eMagiz will periodically look whether (new) files are ready for consumption. There are four key considerations you should take into account when picking up files:
	20
	21	* What are the determining criteria to define when a file is "ready" to be picked up (age, pattern, size, etc.)
	22	* With what frequency is eMagiz going to check for new files
	23	* Has eMagiz the appropriate rights to read from the directory where the messages are to be retrieved from
	24	* What is the format (i.e. XML, JSON, EDI, TXT, CSV, etc.) of the files that eMagiz needs to retrieve
	25
	26	=== 3.1 Determining Criteria ===
	27
	28	By default, eMagiz has a filter that will only accept each file once. This option is called prevent duplicates
	29
17.1	30	[[image:Main.Images.Microlearning.WebHome@crashcourse-messaging-pick-up-files--prevent-duplicates.png]]
1.1	31
	32	As the eMagiz help text clearly states this option will ensure that eMagiz will keep a list in memory and files are only to be passed once.
	33	This check is done before any other filtering is done. So be aware when mixing these settings.
	34
	35	For example, if you combine this setting with an age list filter (i.e. specifying how old a message must be before picking it up) but you pick up messages
	36	at a higher frequency compared to the setting of your age list filter you run the risk of never picking the message up as the message
	37	will be blocked by the age list filter the first time and the second time by the prevent duplicates setting
	38
	39	Another downside of only using the prevent duplicates option is that when you do not delete messages properly and you do restart the flow eMagiz will start picking up those messages again.
	40
	41	So, as you can see determining the proper set of criteria can be quite challenging.
	42	Luckily there is a store component available in the public eMagiz Store that is build based on a set of best practices including the best practice when it comes to filtering.
	43
	44	To filter which messages need to be retrieved from a local directory we use a composite file list filter.
	45
17.1	46	[[image:Main.Images.Microlearning.WebHome@crashcourse-messaging-pick-up-files--composite-file-list-filter-component.png]]
1.1	47
	48	In this filter, we use the following four criteria to filter our messages:
	49
	50	* Regular files only
	51	* Size list filter (minimum 1 bytes, maximum 1048575)
	52	* Age file list filter (minimum 10000)
	53	* Simple pattern file list filter (${file.pickup.pattern})
	54
	55	The result of this configuration should be:
	56
17.1	57	[[image:Main.Images.Microlearning.WebHome@crashcourse-messaging-pick-up-files--composite-file-list-filter.png]]
1.1	58
	59	The next step would be to link this support object to the file inbound channel adapter (the starting point of our flow). To do so open the component and navigate to the Advanced tab.
	60	Here you need to select the correct filter to reference your support object.
	61
17.1	62	[[image:Main.Images.Microlearning.WebHome@crashcourse-messaging-pick-up-files--composite-file-list-filter-linked.png]]
1.1	63
	64	=== 3.2 Frequency ===
	65
	66	Now that we have established the criteria that govern which messages can and which messages cannot pass the next thing
	67	we need to consider is with what frequency we are going to let eMagiz check for new files.
	68
	69	There are three options from which you can choose:
	70
	71	* Fixed Delay
	72	* Fixed-Rate
	73	* Cron Trigger
	74
17.1	75	[[image:Main.Images.Microlearning.WebHome@crashcourse-messaging-pick-up-files--frequency-options.png]]
1.1	76
	77	Both the Fixed Delay and the Fixed Rate are static. This means that when you want to change the frequency with which you look you will have to make a new version of the flow.
	78
	79	The cron trigger can be filled in with the help of a property and can therefore be changed over time without having to change the flow itself.
	80
	81	The best practice is to use the cron trigger in cases where you already expect that the frequency will change over time. If a static value is sufficient the Fixed Delay would be your best option.
	82	The main benefit of the fixed delay above the fixed rate is that it waits before the previous process was finished.
	83	That way you ensure that eMagiz will have to process a huge amount of data in a small amount of time.
	84
	85	=== 3.3 Appropriate Rights ===
	86
	87	To retrieve files from a local directory eMagiz needs to have the appropriate rights to access the directory in question.
	88	If the directory in which the files are located is part of a structure of other directories you need to ensure that eMagiz
	89	has sufficient rights on all directories above the directory from which you want to retrieve files.
	90
	91	If this is not the case you will not be able to retrieve the files from a local directory via eMagiz.
	92
	93	=== 3.4 Message Format ===
	94
	95	Files come in many different forms, shapes, and sizes. Due to the wide variety of options, you will need to ensure that the data is processed correctly.
	96	How to process each message format (i.e. XML, JSON, EDI, CSV, etc.) correctly is explained in other microlearnings where we zoom in on each of the message formats.
	97
21.1	98	For learning about picking up files we need to know that in order for eMagiz to process the content of the file in the onramp and further downstream the contents of the file need to be transformed to string.
1.1	99
	100	There is a component in eMagiz called the file to string transformer. You should place this directly after receiving the file from the external source
	101
17.1	102	[[image:Main.Images.Microlearning.WebHome@crashcourse-messaging-pick-up-files--file-to-string-component.png]]
1.1	103
	104	Configuration wise you need to select the option Delete files to ensure that the files are deleted
	105
17.1	106	[[image:Main.Images.Microlearning.WebHome@crashcourse-messaging-pick-up-files--file-to-string-configuration.png]]
1.1	107
	108	The result of linking the components together will lead to the following solution
	109
17.1	110	[[image:Main.Images.Microlearning.WebHome@crashcourse-messaging-pick-up-files--simple-solution.png]]
1.1	111
	112	With all this in mind, you have successfully learned how to set up your first connectivity via a messaging flow in eMagiz!
	113
22.1	114	== 4. Key takeaways ==
1.1	115
22.6	116	* Criteria for File Retrieval: Determine the criteria for identifying when a file is "ready" to be picked up, such as age, pattern, or size. Ensure these criteria are properly configured to avoid issues with file processing.
	117	* File Formats: Ensure you handle various file formats (e.g., XML, JSON, EDI, CSV) correctly. Use the file to string transformer to process file contents as needed.
	118	* Check Frequency: Decide how frequently eMagiz will check the local directory for new files. You can choose between options like Fixed Delay, Fixed Rate, or Cron Trigger, based on your needs and expected changes in frequency.
	119	* Access Rights: Verify that eMagiz has the necessary permissions to read from the directory and any parent directories. Proper access rights are crucial for successful file retrieval.
1.1	120
22.1	121	== 5. Suggested Additional Readings ==
1.1	122
22.2	123	If you are interested in this topic and want more information on it please read the help text provided by eMagiz and read the following link:
1.1	124
22.5	125	* [[Store (Menu)>>doc:Main.eMagiz Store.WebHome\|\|target="blank"]]
	126	** [[Accelerators (Navigation)>>doc:Main.eMagiz Store.Accelerators.WebHome\|\|target="blank"]]
	127	*** [[File Inbound (Store Item)>>doc:Main.eMagiz Store.Accelerators.File Inbound.WebHome\|\|target="blank"]]
22.4	128	* [[Pick up Files (Search Results)>>url:https://docs.emagiz.com/bin/view/Main/Search?sort=score&sortOrder=desc&highlight=true&facet=true&r=1&f_space_facet=0%2FMain.&l_space_facet=10&f_type=DOCUMENT&f_locale=en&f_locale=&f_locale=en&text=%22pick+up+files%22\|\|target="blank"]]
17.1	129	)))((({{toc/}}))){{/container}}{{/container}}