Changes for page Volume Mapping (On-premise)
Last modified by Erik Bakker on 2024/08/26 12:37
From version 33.1
edited by Erik Bakker
on 2022/06/10 13:46
on 2022/06/10 13:46
Change comment:
There is no comment for this version
To version 28.2
edited by Erik Bakker
on 2022/06/10 13:13
on 2022/06/10 13:13
Change comment:
Update document after refactoring.
Summary
-
Page properties (2 modified, 0 added, 0 removed)
Details
- Page properties
-
- Title
-
... ... @@ -1,1 +1,1 @@ 1 - Archiving1 +novice-file-based-connectivity-processing-a-file-per-line - Content
-
... ... @@ -1,10 +1,10 @@ 1 1 {{container}}{{container layoutStyle="columns"}}((( 2 -In mostcases, the customerwants some kind of insurance policy to determinewhethera filehas entered orleft eMagiz. Onewayof supplyingsuch functionalityfor file-baseddataexchangeis throughthe use of archivingfunctionality.Withthe help of this functionalitycanyoueasilywrite thefileas received (or about to be sent)toa separatelocation. By ensuring that the datastaysin that location for a certain amountof timeand by giving thecustomerand yourselfaccessyou have asortof audit trail that details themessagesthat havebeen exchanged. This archivecouldalsobeused toanalyzeproblemsincasethings gowrong. In this microlearning,we will learn howyoucansetup that archivingandlearnhow to clean upthearchiving toensurethatdataisonly keptfor a limited period.2 +In this microlearning, we will learn how you can define a header line in which you specify the naming of the various columns. Some external systems require a header line when you supply them with data via a flat file that is placed somewhere. 3 3 4 4 Should you have any questions, please contact [[academy@emagiz.com>>mailto:academy@emagiz.com]]. 5 5 6 -* Last update: May 31th, 20217 -* Required reading time: 6minutes6 +* Last update: May 28th, 2021 7 +* Required reading time: 5 minutes 8 8 9 9 == 1. Prerequisites == 10 10 ... ... @@ -12,81 +12,59 @@ 12 12 13 13 == 2. Key concepts == 14 14 15 -This microlearning centers around learning how to a rchivecorrectly.15 +This microlearning centers around learning how to place a header line on a flat-file output. 16 16 17 -By ar chiving,we mean:Temporarilystoringdatafor auditrposes andpossibleretry scenarios.17 +By header line we mean: A line in the output that defines the naming of the various columns 18 18 19 -* Archiving is used for audit purposes 20 -* Archiving is used for retry scenarios 21 -* Ensure that data is cleaned after a retention period to keep in control of the data 19 +Some external parties require that the first line in the flat file output (i.e. CSV) is filled with column names (i.e. headers). In eMagiz, we call this line a header line. 22 22 23 -== 3. Archiving==21 +== 3. Header line == 24 24 25 -In mostcases,thecustomer wants some kindofinsurancepolicyto determinewhetherafilehasenteredorleft eMagiz. Oneway of supplyingsuchfunctionalityforfile-baseddata exchangeis throughtheuse ofarchiving functionality.With thehelp ofthis functionalitycanyoueasilywritethe file asreceived(orabout to besent)toaseparatelocation. Byensuringthatthedatastaysin thatlocationfor a certain amountof timeandby giving the customerand yourselfaccessyouhavesortof audittrail that details themessagesthathave beenxchanged.Thisarchivecouldalso be usedtoanalyzeproblemsin casethingsgo wrong. Inthismicrolearning,wewill learn how youcansetup that archivingand learn how to cleanupthearchiving toensure thatdatais only keptforalimited period.23 +In this microlearning, we will learn how you can define a header line in which you specify the naming of the various columns. Some external systems require a header line when you supply them with data via a flat file that is placed somewhere. The header line is the first line in the flat file output. Within this line, the various column names are specified for clarity. 26 26 27 - ===3.1Archiving itself===25 +To add such a header line in eMagiz you need to navigate to the Create phase of eMagiz and open the exit flow in which you want to drop the file to a certain location. Within the context of this flow, we need to add functionality that will ensure that a header line is written to the output before any functional lines are added. To do so first enter "Start Editing" mode on flow level. After you have done so please add a file outbound channel adapter to the flow including an input channel. We will use this component to write our header line to the flat file output. 28 28 29 - To make thiswork ineMagizyou needto navigateto theCreate phaseof eMagiz and open the entryflowinwhich youwant to archivethefiles. Within thecontext of this flow, we need to add functionality that willensurethat eachinput file isarchived and cleaned up when olderthan threedays. To do so first enter "Start Editing"modeon flow level. The firstdecisionwe have totake is how we aregoing to name thefiles withinthe archiving. The bestpractice, in this case, is theoriginal filename+ the currenttime as a suffix. Youcanefinethis by dragging a format filenamegenerator (supportobject) to the canvas.27 +Ensure that the directory to which you reference is the same as in your functional file outbound channel adapter. 30 30 31 -[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-ar chiving--file-name-generator.png]]29 +[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-header-line--file-outbound-channel-header-line.png]] 32 32 33 - Afterwehavedonethispleaseaddafile outbound channeladapter totheflowincludinganinputchannel.Ensurethat you use apropertyfor thedirectorythatreferencesanotherdirectory comparedtotheinput directorytopreventcreatinganinfiniteloop.31 +Now it is time to turn our attention to the Advanced tab. For the Mode select Ignore. Select this option to ensure that the header line is only written down once when there is no output created yet and not somewhere in the middle, in the end, or every time. Furthermore, select the option Append New Line to ensure that the remainder of the information is not appended to the same line. 34 34 35 -[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity- archiving--archiving-config-file-outbound-basic.png]]33 +[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-header-line--file-outbound-channel-header-line-advanced.png]] 36 36 37 - Nowthat we haveconfiguredthe basicsletus turn ourattentiontotheadvancedconfiguration.Intheadvanced tab of this component,weneedtoselect the file namegeneratortoensurethatthefilesenamedcorrectly.Incaseyou processeachlineseparatelyyouhavetochoosewhethertosavethemasseparatefilesinthearchiveorby appendingthem again.This canbe achievedbyselecting thecorrectMode.Inmostcases,however,thedefaultMode of Replacewilluffice.35 +After you have done so we need to add a standard transformer that defines the various column names to be written to the flat file output. To do so add the standard transformer component to the canvas including an input channel. After you have done so define the relevant SpEL expression. In this case, we advise using a property value that represents a string of column names. The value of the property should be something as follows: 38 38 39 - [[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-archiving--archiving-config-file-outbound-advanced.png]]37 +'Header1;Header2;Header3;Header4' 40 40 41 - The momentyou are satisfiedpress Save. Nowthatwehaveconfiguredthisit becomestimeto determinehow weget the neededinputtowrite to ourarchive.Intheexample we aresing herewewanttoarchiveour input fileso we needtoensure thatthe data we received iswrittentothearchive as soon as possible. Todo so place awiretaponthefirst channel after retrievingthe file. This willmake surethat themessage isarchivedbefore processedfurther.The result shouldbe something as shown below. Notethat thissame piece oflogic could be applied inother flowswithin the eMagiz platform in a similar manner.39 +Do note that the separator, in this case, needs to match the requirements of the external system. At the flow configuration level, the standard transformer should look as follows. 42 42 43 -[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-ar chiving--archiving-result.png]]41 +[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-header-line--define-columns-names.png]] 44 44 45 - ===3.2Clean up theArchive===43 +Our last step is to ensure that this piece of logic is tied to the main flow and is executed before writing the functional line(s) to the output file. To make that happen we need to add a wiretap to the flow. With the help of this functionality, you can define which part of the logic takes precedence over another part of the logic. To do so double click on the channel on which you want to place a wiretap, select the option wiretap and select the correct wiretap channel. After you have done this the result should be something as follows: 46 46 47 - To ensure that thedataisnot keptindefinitely we need toclean up the archive.doso to prevent problems with disk spacebut also to prevent data leaks ofold data that couldimpact theprivacy of others. Before wecan set up the logic ineMagiz weneed to talk to thecustomerto see what an acceptabletermis withinhich thedatais kept. In most cases, this is a week ortwo weeks. In this example, we have chosenthree days.45 +[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-header-line--wiretap-result.png]] 48 48 49 - Nowthatwe knowthelimit it istimetoconfigure thecomponents.Westart withcompositeilefilter(supportobject).Withinthisfilter, we atleastdefinehow old a filemustbe beforeit canbe deleted (in milliseconds). Ifwe turnhree daysintomillisecondswe get 259200000. Furthermore,we at least definethatweonlywanttodeleteregularfiles.47 +With these couple of steps, you have now successfully added logic to your flow that will ensure that a header line is added before any functional line(s) are written to the output file. 50 50 51 -[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-archiving--file-list-filter-for-archive-cleanup.png]] 52 - 53 -Having done so we can add a file inbound channel adapter to the canvas including an output channel. Ensure that the property reference for the directory matches the one you have used before in the outbound channel adapter. Furthermore link the filter to the component and define the poller according to the best practice. 54 - 55 -[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-archiving--file-inbound-archive-cleanup.png]] 56 - 57 -One thing we should not forget within this configuration is to set the Max messages per poll on the Advanced tab of the poller-configuration to a sufficiently high number (i.e. 50). If you forget to do so and you only check once a day it will mean that only one message will be deleted that day. 58 - 59 -[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-archiving--file-inbound-archive-cleanup-max-messages-per-poll.png]] 60 - 61 -Now eMagiz will check on a set time interval whether there are files that are older than three days that are ready for deletion. One last step to go. This last step will ensure that all files that fit the bill will be deleted from the archive. Simply add a standard service activator to the canvas and define the following SPeL expression within the component: payload.delete(). 62 - 63 -[[image:Main.Images.Microlearning.WebHome@novice-file-based-connectivity-archiving--archive-cleanup-deletion.png]] 64 - 65 -This will ensure that each file that is retrieved will indeed be deleted from the archive. 66 - 67 67 == 4. Assignment == 68 68 69 -Configure an e ntryin which youbuildthearchivingand thecleanupofthe archiving.51 +Configure an exit in which you define and write a header line to a flat-file output before adding functional lines. 70 70 This assignment can be completed with the help of the (Academy) project that you have created/used in the previous assignment. 71 71 72 72 == 5. Key takeaways == 73 73 74 -* Archiving is used for audit purposes 75 -* Archiving is used for retry scenarios 76 -* Ensure that data is cleaned after a retention period to keep in control of the data 77 -* Don't forget the max messages per poll 56 +* The header line contains the names of the columns of the flat file output 57 +* Use the Ignore mode to ensure the header line is created once 58 +* Use the wiretap to ensure the header line is created first 78 78 79 79 == 6. Suggested Additional Readings == 80 80 81 - Ifyouareinterestedin this topicandwant moreinformationon it pleasereadthe help text provided by eMagizandcheck outthe followingstorecontent:62 +There are no suggested additional readings on this topic 82 82 83 -* [[microlearning>>doc:Main.eMagiz Store.Accelerators.File Archiving.WebHome||target="blank"]] 84 -* [[microlearning>>doc:Main.eMagiz Store.Accelerators.Delete Folder's.WebHome||target="blank"]] 85 - 86 86 == 7. Silent demonstration video == 87 87 88 88 This video demonstrates how you could have handled the assignment and gives you some context on what you have just learned. 89 89 90 -{{video attachment="novice-file-based-connectivity- characterset.mp4" reference="Main.Videos.Microlearning.WebHome"/}}68 +{{video attachment="novice-file-based-connectivity-header-line.mp4" reference="Main.Videos.Microlearning.WebHome"/}} 91 91 92 92 )))((({{toc/}}))){{/container}}{{/container}}