Debugging with the Core Analysis Loop
When you receive a notification, either from an alert or directly from a user, that something in your application is not working correctly, you know that something is wrong but you do not know why.
Following the core analysis loop, as introduced in the book Observability Engineering, allows you to use telemetry “to form hypotheses and to validate or invalidate them with data, and thereby systematically arrive at the answer to a complex problem” (see Figure 8-10).
Figure 8-10. The core analysis loop
Start by verifying the problem that has been reported to you by analyzing the health of the surrounding systems and services. Use your critical health dashboard to spot curves, spikes, or dips in metric graphs. Find the outlying data points or events causing this deviation from stable performance and identify patterns that could point to a root cause. Next, go back to the wider observability data (traces, logs, metrics) and filter by the pattern to validate your hypothesis. If you are confident the root cause has been found, you can attempt to remedy it. Otherwise, if the problem remains undiagnosed, start the analysis loop again to refocus your investigation.
Always try to resist reactionary conclusions like “That’s always breaking,” “You can’t rely on third party X,” “We’ve seen this before,” and “It’s probably due to Y.” This can result in missing root causes and general apathy toward operational tasks. The core analysis loop helps your engineers to debug from first principles.
Disaster Recovery
Architecting and operating your serverless application to be tolerant of faults cannot guard against all types of failures. You will have parts of your system that represent a single point of failure that could cause entire critical paths to break in the event of a fault. The uptime of your application is also subject to the status of the third-party software services you depend on, including AWS.