Skip to Content
Last modified by Erik Bakker on 2024/09/16 07:01
Show last authors
1 {{container}}{{container layoutStyle="columns"}}(((
2 In this microlearning, we'll guide you through the essential steps to analyze and respond to system alerts that may indicate a need for recovery actions. We’ll cover various scenarios, such as server downtime or message traffic issues, and provide clear instructions to help you resolve these problems efficiently. By understanding these key signals and how to address them, you'll be better equipped to maintain system stability and minimize downtime.
3
4 Should you have any questions, please contact [[academy@emagiz.com>>mailto:academy@emagiz.com]].
5
6 == 1. Prerequisites ==
7
8 * Expert knowledge of the eMagiz platform
9 * Received a signal that something is going wrong
10
11 == 2. Key concepts ==
12
13 In this microlearning, we have detailed how to analyze whether recovery is needed and the steps you need to take based on certain signals.
14
15 * We will discuss various alerts (i.e., signals) you can receive that might require action.
16 * Per alert, we will determine how you can best execute your analysis.
17 * If applicable, we detail what to do when your actions prove unsuccessful.
18
19 == 3. Signal received - Recovery Needed ==
20
21 In this microlearning we have detailed how you can analyze whether recovery is needed. On top of that we have detailed the steps you need to take based on certain signals.
22
23 We will discuss the following signals:
24 * No messages can be send to system X
25 * Live server seems to be down
26 * Backup server does not start up after failback scenario
27 * Connector / Runtime seems to be down
28 * Number of consumers is less than threshold
29 * Out of memory log entry
30
31 === 3.1 No traffic reported related to the runtime for which you have received the alert ===
32
33 1. Log in to eMagiz.
34 1. Open the bus for which you have received the alert, navigate to ‘Manage’ and select the ‘Production’ environment.
35 1*. [[Check runtime statistics>>||anchor="H3.9Checkruntimestatistics"]].
36 1. Determine how to proceed.
37 1*. Runtime statistics indicate nothing is happening for some time now
38 1**. [[Check messages in queue>>||anchor="H3.10Checkmessagesinqueue"]].
39 1**. [[Check runtime overview>>||anchor="H3.11Checkruntimeoverview"]]. Determine whether the relevant runtime (i.e. **Container**) is active.
40 1***. Yes -> Continue with step 4.
41 1***. No -> In case you are dealing with a failover bus, check whether the backup JMS Server is running, by executing the process as described above.
42 1****. Yes -> Backup should handle the message traffic. Please check out the [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]] section for what to do next.
43 1****. No -> [[Restart live and/or backup server>>||anchor="H3.8Restartliveand2Forbackupserver"]].
44 1*. Runtime statistics indicate that statistics are coming in.
45 1**. Continue with step 4
46 1. Confirm messages are being consumed from the queues and verify in the application itself (by yourself or with the help from someone else) whether messages arrive and are being processed. For help, see [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]].
47 1. In case the above does not explain why the alert was triggered please check several other models running the same architecture to verify whether they experience similar problems ([[Messages missing for multiple environments>>||anchor="H3.14Checkwhetherstatisticsaremissingformultiplebusenvironments"]]). If so, please contact support for assistance.
48
49 === 3.2 Both live JMS as well as backup JMS are **reportedly** down ===
50
51 {{warning}}Note that you can also receive this alert in a **single lane** setup. In that instance the alert is only focused on the **live** JMS as there is no backup JMS available in a **single lane** setup.{{/warning}}
52
53 1. Log in to eMagiz
54 1. Open the bus for which you have received the alert, navigate to ‘Manage’ and select the ‘Production’ environment
55 1*. [[Check runtime statistics>>||anchor="H3.9Checkruntimestatistics"]].
56 1. Determine how to proceed.
57 1*. Runtime statistics indicate nothing is happening for some time now
58 1**. [[Check messages in queue>>||anchor="H3.10Checkmessagesinqueue"]].
59 1**. [[Check runtime overview>>||anchor="H3.11Checkruntimeoverview"]]. Determine whether the relevant runtime (i.e. **JMS Server**) is active.
60 1***. Yes -> Continue with step 4.
61 1***. No -> In case you are dealing with a failover bus, check whether the backup JMS Server is running, by executing the process as described above.
62 1****. Yes -> Backup should handle the message traffic. Please check out the [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]] section for what to do next.
63 1****. No -> [[Restart live and/or backup server>>||anchor="H3.8Restartliveand2Forbackupserver"]].
64 1*. Runtime statistics indicate that statistics are coming in.
65 1**. Continue with step 4
66 1. Confirm messages are being consumed from the queues and verify in the application itself (by yourself or with the help from someone else) whether messages arrive and are being processed. For help, see [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]].
67 1. In case the above does not explain why the alert was triggered please check several other models running the same architecture to verify whether they experience similar problems ([[Messages missing for multiple environments>>||anchor="H3.14Checkwhetherstatisticsaremissingformultiplebusenvironments"]]). If so, please contact support for assistance.
68
69 === 3.3 Backup server won’t come up again after restarting the **live** JMS ===
70
71 {{warning}}This scenario is **only** applicable for buses with a **failover setup**.{{/warning}}
72
73 1. Log in to eMagiz
74 1. Open the bus for which you have received the alert, navigate to ‘Manage’ and select the ‘Production’ environment
75 1*. Navigate to the sub tab ‘Monitoring’ and select the ‘[[Log Entries>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Active Monitoring.intermediate-active-monitoring-understanding-log-entries||target="blank"]]’ option in the left hand panel.
76 1**. Check whether a certain logging line is present in the eMagiz logs. Search for each of the following values in the field message via separate searches: ‘Java heap space’ or ‘Out of memory’ or ‘I/O error’ or ‘Metaspace error’
77 1***. Yes -> Inform customer that the backup is currently not working and discuss with your colleagues, support and the customer for a suitable time window to get the backup up and running again.
78 1***. No -> Continue with step 3.
79 1. IIn case you reach this step, verify that the number of consumers on a container runtime is **two**.
80 1*. Yes -> No further action required.
81 1*. No -> Restart backup server. [[Restart live and/or backup server>>||anchor="H3.8Restartliveand2Forbackupserver"]]
82
83 === 3.4 Runtime is down ===
84
85 1. Log in to eMagiz
86 1. Open the bus for which you have received the alert, navigate to ‘Manage’ and select the ‘Production’ environment
87 1*. [[Check runtime statistics>>||anchor="H3.9Checkruntimestatistics"]].
88 1. Determine how to proceed.
89 1*. Runtime statistics indicate nothing is happening for some time now
90 1**. [[Check messages in queue>>||anchor="H3.10Checkmessagesinqueue"]].
91 1**. Determine whether the check you have executed as part of the previous step indicates that the runtime is **down**.
92 1***. Yes -> Runtime should be restarted
93 1****. Navigate to ‘Deploy’, ‘Architecture’ and select the ‘Production’ environment. Search for the runtime that is not working anymore.
94 1****. Determine on the basis of the search where the connector is running.
95 1*****. In case of a cloud connector you can restart the connector via this page. [[Restart a runtime>>||anchor="H3.6Restartaruntime"]]
96 1******. Restart **not** successful -> Contact support for assistance as they can check the logs in the eMagiz Cloud.
97 1******. Restart successful -> No further action needed.
98 1***. No -> Temporary connection loss between eMagiz cloud and runtime. If message traffic works as expected consider this an incident and log an RCA with support.
99 1*. Runtime statistics indicate that statistics are coming in
100 1**. Confirm that messages are indeed consumed from the queue and check whether messages arrive and are being processed
101 1***. Messages arrive -> Continue with step 4.
102 1***. Messages do not arrive -> Continue with the step below.
103 1****. In case your runtime statistics are coming in but no messages are consumed and delivered please see [[Solve the problem where there is no traffic between eMagiz and the runtime for which you have received the alert>>||anchor="H3.1Notrafficreportedrelatedtotheruntimeforwhichyouhavereceivedthealert"]].
104 1. Log an RCA with support for further analysis
105
106 === 3.5 Too few consumers reported ===
107
108 1. Did you receive an alert that there are too few consumers on queue?
109 1*. Yes -> Navigate to ‘Manage’ and select ‘[[Queue Statistics>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Platform.crashcourse-platform-runtime-overview||target="blank"]]’ on the ‘Production’ environment from the left hand panel. Check various flows that are running on this runtime. The queue statistics should indicate that the number of consumers have been dropped from **2** to **1** or from **1** to **0**.
110 1**. Success -> This means that this specific runtime is indeed down. See [[Restart a runtime>>||anchor="H3.6Restartaruntime"]] to restart the runtime in question.
111 1***. Check whether the number of consumers is back to the expected level
112 1****. Yes -> Problem solved
113 1****. No -> Potentially there are bigger issues on JMS Server level. Navigate to [[Solve the problem where both live JMS as well as backup JMS are down>>||anchor="H3.2BothliveJMSaswellasbackupJMSarereportedlydown"]].
114 1**. Failure -> This means the consumer count works as expected. Consider this an Incident.
115
116 === 3.6 Restart a runtime ===
117
118 1. Navigate to ‘Deploy’, ‘Architecture’ and select the ‘Production’ environment if you are not yet on this page
119 1*. Press Start Editing
120 1. Search for the runtime for which you have received the alert and activate the drop down menu via a right click of your mouse
121 1*. Select **[[Restart Runtime>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.eMagiz Runtime Management.intermediate-emagiz-runtime-management-restart-runtime.WebHome||target="blank"]]**.
122 1. Check traffic of messages, see [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]].
123 1*. Success -> Tell customer that messages are once again delivered correctly
124 1*. Failure -> [[Check whether runtime exists in eMagiz Cloud>>||anchor="H3.13Checkwhetherruntimeisactuallydeployed"]].
125
126 === 3.7 Reset a runtime ===
127
128 1. Navigate to ‘Deploy’, ‘Architecture’ and select the ‘Production’ environment if you are not yet on this page
129 1*. Press Start Editing
130 1. Search for the runtime for which you have received the alert and activate the drop down menu via a right click of your mouse
131 1*. Select **[[Reset Runtime>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.eMagiz Runtime Management.intermediate-emagiz-runtime-management-reset-runtime.WebHome||target="blank"]]**.
132 1. Check traffic of messages, see [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]].
133 1*. Success -> Tell customer that messages are once again delivered correctly
134 1*. Failure -> [[Check whether runtime exists in eMagiz Cloud>>||anchor="H3.13Checkwhetherruntimeisactuallydeployed"]].
135
136 === 3.8 Restart live and/or backup server ===
137
138 1. Navigate to ‘Deploy’ and select the ‘Production’ environment.
139 1*. [[Check runtime overview>>||anchor="H3.11Checkruntimeoverview"]]. Determine whether the relevant runtime (i.e. **JMS Server**) is active.
140
141 {{info}}If needed, execute the same action for the backup server (if this is present with your customer).{{/info}}
142
143 {{warning}}To proceed in restarting JMS runtimes we offer two alternative paths.
144 * Option A defines the least risky option. Opting for this is however the most time consuming option.
145 * Option B defines the fast option. Opting for this is however the most risky option.{{/warning}}
146
147 ==== 3.8.1 Option A ====
148
149 Are both the live and backup server down? The following steps describe the least risky option, but is also the most time consuming option.
150
151 1. Stop backup server
152 1*. Go to ‘Deploy’, ‘Architecture’ and press Start Editing
153 1*. Right mouse click on the machine where the backup JMS runtime is running on and select Stop machine
154 1. Stop live server
155 1*. Go to ‘Deploy’, ‘Architecture’ and press Start Editing
156 1*. Right mouse click on the machine where the live JMS runtime is running on and select Stop machine
157 1. Start live server
158 1*. Go to ‘Deploy’, ‘Architecture’ and press Start Editing
159 1*. Right mouse click on the machine where the live JMS runtime is running on and select Start machine
160 1. Start backup server
161 1*. Go to ‘Deploy’, ‘Architecture’ and press Start Editing
162 1*. Right mouse click on the machine where the backup JMS runtime is running on and select Start machine
163 1. [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]]
164 1*. Success -> Communicate to the customer that messages arrive again
165 1*. Failure -> Contact support for assistance as they can check the logs in the eMagiz Cloud
166
167 ==== 3.8.1 Option B ====
168
169 If uptime and speed in which actions are executed are of importance please follow the steps detailed below.
170
171 1. Restart the JMS (live) runtime
172 1*. Navigate to ‘Deploy’, ‘Architecture’ and press Start Editing
173 1*. Right mouse click on the JMS runtime and select Restart Runtime
174 1. Restart the JMS (backup) runtime
175 1*. Navigate to ‘Deploy’, ‘Architecture’ and press Start Editing
176 1*. Right mouse click on the JMS runtime and select Restart Runtime
177 1. [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]]
178 1*. Success -> Communicate to the customer that messages arrive again
179 1*. Failure -> Contact support for assistance as they can check the logs in the eMagiz Cloud
180
181 ===== 3.8.1.1 Double Lane variant =====
182
183 {{warning}}The following step is only applicable for double lane configurations.{{/warning}}
184
185 Is live server down but is the backup running or vice versa?
186
187 1. Stop live server or backup server (depending on which of the two are down)
188 1*. Navigate to ‘Deploy’, ‘Architecture’ and press Start Editing
189 1*. Right mouse click on the JMS runtime and select Stop Runtime
190 1. Start live server or backup server (depending on which of the two are down)
191 1*. Navigate to ‘Deploy’, ‘Architecture’ and press Start Editing
192 1*. Right mouse click on the JMS runtime and select Start Runtime
193 1. Check runtime statistics under ‘Manage’
194 1*. Success -> Communicate to the customer that both servers are up and running again
195 1*. Failure -> [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]]
196 1**. Success -> Open an RCA for the project team and or support to identify why the problem occurred.
197 1**. Failure -> Contact support for assistance as they can check the logs in the eMagiz Cloud.
198
199 === 3.9 Check runtime statistics ===
200
201 1. Navigate to the sub tab ‘Monitoring’ and select the ‘[[Runtime Statistics>>doc:Main.eMagiz Academy.Microlearnings.Advanced Level.Advanced monitoring.advanced-monitoring-runtime-statistics-key-metrics.WebHome||target="blank"]]’ option in the left hand panel.
202 1*. Check for all relevant runtimes whether the runtime statistics are indeed missing.
203
204 [[image:Main.Images.Microlearning.WebHome@expert-recovery-guide-signalx-reproducable-steps--runtime-statistics-overview.png]]
205
206 1. Check for the last measured point in time to determine whether this time matches the current time **in UTC**.
207
208 === 3.10 Check messages in queue ===
209
210 1. Navigate to ‘Manage’, ‘Monitoring’ and select ‘[[Queue Statistics>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Platform.crashcourse-platform-runtime-overview||target="blank"]]’ in the left hand panel.
211 1*. Check whether there are a lot of messages residing in queues towards this system.
212
213 {{info}}Note that entries don't have a corresponding queue{{/info}}
214
215 [[image:Main.Images.Microlearning.WebHome@expert-recovery-guide-signalx-reproducable-steps--queue-statistics-overview.png]]
216
217 === 3.11 Check runtime overview ===
218
219 1. Navigate to ‘Deploy’, ‘[[Runtime Overview>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Messaging.crashcourse-messaging-interpreting-queue-statistics-gen3||target="blank"]]’.
220 1*. Verify whether the runtime in question is **Active**. This can be done by searching for a keyphrase (i.e "jms-server") and verifying whether the "state" is **Running**.
221
222 [[image:Main.Images.Microlearning.WebHome@expert-recovery-guide-signalx-reproducable-steps--runtime-overview.png]]
223
224 === 3.12 Check message traffic ===
225
226 1. Navigate to ‘Manage’ and select the option ‘[[Queue Statistics>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Platform.crashcourse-platform-runtime-overview||target="blank"]]’ in the left hand panel. Check the various flows that are of relevance for this particular integration. For this integration you should see messages flowing through each step. Be aware, the number of messages does not have to be equal in each step due to the possibility of filtering messages in between.
227 1*. Success -> This means everything works again as expected. If you happen to have access to the application that should receive the data you always have the extra option to log in and verify if the messages indeed have arrived
228 1*. Failure -> This means that messages are still not being delivered. In most cases this is due to problems on JMS level. Advice is to restart the JMS runtime. See [[Restart live and/or backup server>>||anchor="H3.8Restartliveand2Forbackupserver"]].
229
230 === 3.13 Check whether runtime is actually deployed ===
231
232 1. Navigate to ‘Deploy’, ‘Architecture’ and select the ‘Production’ environment if you are not yet on this page
233 1. Search for the runtime that is not running and see if you can find it
234 1*. Yes, check the background color of the runtime and proceed with step 3
235 1*. No, navigate to ‘Design’, ‘[[Architecture>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Platform.crashcourse-platform-design-understanding-design-architecture-basic||target="blank"]]’. Select ‘Production’ and continue with step 6
236 1. Background color of the runtime is
237 1*. White with a green outside line. Continue with step 4
238 1*. White with a blue outside line. Continue with step 4
239 1*. White with a red outside line. Continue with step 5
240 1*. Dark blue with a dark blue outside line -> [[Reset a runtime in AWS>>||anchor="H3.7ResetaruntimeinAWS"]].
241 1. If you arrived at this step this means that a change to Architecture has not yet been committed to the eMagiz Cloud. To commit these changes execute the following steps in order
242 1*. Press Start Editing
243 1*. Press Apply to Environment and wait for the conformation from eMagiz that the update is committed to the eMagiz Cloud
244 1*. [[Check message traffic>>||anchor="H3.12Checkmessagetraffic"]]
245 1**. If this control renders success your runtime is (again) running as expected
246 1**. If this control does not render success please contact support to verify what the logs within the eMagiz Cloud tell with regards to why this runtime is not active
247 1. If you arrived at this step this means that the runtime that is not running actually needs to be removed from the eMagiz Cloud
248 1*. If this is indeed true, continue with step 4
249 1*. If this is not correct navigate to ‘Design’, ‘[[Architecture>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Platform.crashcourse-platform-design-understanding-design-architecture-basic||target="blank"]]’ and select ‘Production’. Please continue with step 6 afterwards
250 1. If you arrived at this step you have arrived at the conclusion that the Architecture is not conform what actually should be running on your environment. In these cases you need to execute the following actions
251 1*. Press Start Editing
252 1*. Press Apply to Environment
253 1*. Place runtime on the machine it should be running on
254 1*. Press Stop Editing
255 1*. Navigate back to step 2 and follow the steps from there
256
257 === 3.14 Check whether statistics are missing for multiple bus environments ===
258
259 1. Select a number of other buses and verify under ‘Manage’ whether those buses do have statistics (queue, runtime, etc.)
260 1*. Success -> Problem resides with the bus you are currently investigating. Restart of the JMS server is required. If the problem is only that statistics are missing please consult with the customer first before executing a restart of the environment
261 1*. Failure -> Contact the support department.
262
263 == 4. Key takeaways ==
264
265 * With this microlearning you can better analyze problems and recover your instances with little downtime
266
267 == 5. Suggested Additional Readings ==
268
269 If you are interested in this topic and want more information on it please read the help text provided by eMagiz and read the following link:
270
271 * [[Crash Courses (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.WebHome||target="blank"]]
272 ** [[Crash Course Messaging (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Messaging.WebHome||target="blank"]]
273 *** [[Interpreting Queue Statistics (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Messaging.crashcourse-messaging-interpreting-queue-statistics-gen3||target="blank"]]
274 * [[Fundamentals (Menu)>>doc:Main.eMagiz Academy.Fundamentals.WebHome||target="blank"]]
275 ** [[Traceability in eMagiz (Menu)>>doc:Main.eMagiz Academy.Fundamentals.fundamental-traceability-in-emagiz||target="blank"]]
276 * [[Crash Course (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.WebHome||target="blank"]]
277 ** [[Crash Course Platform (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Platform.WebHome||target="blank"]]
278 *** [[Interpreting Runtime Statistics (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Crash Course.Crash Course Platform.crashcourse-platform-manage-interpreting-runtime-statistics-gen3.WebHome||target="blank"]]
279 * [[Intermediate (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.WebHome||target="blank"]]
280 ** [[DevOps Perspectives (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.DevOps Perspectives.WebHome||target="blank"]]
281 *** [[Daily Monitoring (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.DevOps Perspectives.intermediate-devops-perspectives-daily-monitoring||target="blank"]]
282 ** [[eMagiz Runtime Management (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.eMagiz Runtime Management.WebHome||target="blank"]]
283 *** [[Restart Runtime (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.eMagiz Runtime Management.intermediate-emagiz-runtime-management-restart-runtime.WebHome||target="blank"]]
284 ** [[Active Monitoring>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Active Monitoring.WebHome||target="blank"]]
285 *** [[Understanding Log Entries>>doc:Main.eMagiz Academy.Microlearnings.Intermediate Level.Active Monitoring.intermediate-active-monitoring-understanding-log-entries||target="blank"]]
286 * [[Advanced (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Advanced Level.WebHome||target="blank"]]
287 ** [[Monitoring (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Advanced Level.Advanced monitoring.WebHome||target="blank"]]
288 *** [[Runtime Statistics - Key metrics (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Advanced Level.Advanced monitoring.advanced-monitoring-runtime-statistics-key-metrics.WebHome||target="blank"]]
289 *** [[HTTP Statistics (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Advanced Level.Advanced monitoring.advanced-monitoring-http-statistics.WebHome||target="blank"]]
290 ** [[Lifecycle Management (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Advanced Level.Lifecycle Management.WebHome||target="blank"]]
291 *** [[Impact of Runtime Image Upgrades (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Advanced Level.Lifecycle Management.advanced-lifecycle-management-impact-of-runtime-image-upgrades.WebHome||target="blank"]]
292 * [[Expert (Menu)>>doc:Main.eMagiz Academy.Microlearnings.Expert Level.WebHome||target="blank"]]
293 ** [[Monitoring (Navigation)>>doc:Main.eMagiz Academy.Microlearnings.Expert Level.Monitoring.WebHome||target="blank"]]
294 *** [[Runtime Metrics - Other metrics (Explanation)>>doc:Main.eMagiz Academy.Microlearnings.Expert Level.Monitoring.expert-monitoring-runtime-statistics-other-metrics||target="blank"]]
295 * [[Recovery (Search Result)>>url:https://docs.emagiz.com/bin/view/Main/Search?sort=score&sortOrder=desc&highlight=true&facet=true&r=1&f_space_facet=0%2FMain.&f_type=DOCUMENT&f_locale=en&f_locale=&f_locale=en&text=recovery||target="blank"]]
296 )))((({{toc/}}))){{/container}}{{/container}}