Part 10. How to Monitor Your Systems

Broadly speaking, you can categorize metrics into the following buckets:

Let’s dig into each of these.

To make use of all the kinds of metrics you just saw, there are three steps:

Some monitoring tools are designed to do all three of these steps, while others are focused on just one or two.

In this section, you’ll use CloudWatch, a monitoring tool built into AWS, to get metrics for EC2 instances, ASGs, and ALBs. Out of the box, CloudWatch provides application and server metrics for most AWS resources; you’ll also deploy Route 53 health checks to get availability metrics. Finally, you’ll gather all these metrics in a custom CloudWatch dashboard.

To get started, head into the folder you’ve been using for all the examples in this blog post series and create a new ch10/tofu/live folder:

Next, copy your asg-sample module from Part 3 into the tofu/live folder:

The asg-sample module deploys an ASG with an ALB in front of it. It’s configured to deploy an AMI built from the Packer template in ch3/packer (this is the sample app before you added TLS code to it in Part 8). If you still have that AMI, you can reuse it; if not, see Section 3.3.1 for how to build it again.

Next, add a Route 53 health check in main.tf, as shown in Example 190.

This code configures a Route 53 health check that will test your service from multiple endpoints around the world, with the following configuration:

Next, you can create a CloudWatch dashboard by using a module called cloudwatch-dashboard, which is in the blog post series’s sample code repo in the ch10/tofu/modules/cloudwatch-dashboard folder. This module creates a dashboard that highlights a few specific metrics for the ASG, ALB, and Route 53 health checks. Example 191 shows how to update main.tf to use the cloudwatch-dashboard module.

The preceding code configures the dashboard as follows:

Finally, add the dashboard URL as an output variable in outputs.tf, as shown in Example 192.

Deploy as usual, authenticating to AWS, and running init and apply:

It will take 3–10 minutes for the ASG to deploy and for the apps to boot up, so be patient. When everything is done, you should see two output variables:

Test the URL in the alb_dns_name output to check that the app is working (make sure to use HTTP, as this app from back in Part 3 does not support HTTPS):

You should see the usual Hello, World! text. Next open up the URL in the dashboard_url output, and you should see a dashboard similar to Figure 95.

If the dashboard is empty, wait a few minutes for the Route 53 health checks to make requests to your app, and you should start seeing the following metrics:

Congrats, you’re now seeing a small set of useful application, server, and availability metrics! If you want to see all available metrics in CloudWatch, head over to the CloudWatch metrics dashboard; if you want to see more info on the availability metrics, head over to the Route 53 health checks dashboard.

When you’re done testing, you may wish to run tofu destroy to clean up your infrastructure so you don’t accumulate any charges. Alternatively, you may want to wait until later in this blog post, when you update this example code to enable alerts in CloudWatch. Either way, make sure to commit your latest code to Git.

Let’s now move on to a discussion of structured events and how they compare to other types of monitoring you’ve seen so far.

As your architecture becomes more complicated, and it evolves along the lines you saw in Section 1.4 (e.g., you move from a monolith to microservices, from single-node systems to distributed systems, and so on) two things will start to happen:

By the time you get to the scale of massive companies like Google and Meta, you’ll constantly be dealing with problems for which you don’t have, and never would’ve thought to have, automated tests, metrics, or dashboards. While you can anticipate that it’s worth monitoring the four golden signals (latency, traffic, errors, and saturation) in almost any company, how do you deal with complex, unanticipated, and even unimaginable problems? How do you deal with the unknown unknowns? This is where observability comes into the picture.

Observability is a property of a system. In control theory, it’s a measure of how well the internal states of that system can be inferred from knowledge of its external outputs. In DevOps, it’s a measure of how well you can understand any state your software may have gotten itself into, solely by interrogating that software with external tools. That means you can figure out what caused a problem, no matter how weird or unpredictable, without shipping new code, but solely through an ad hoc, iterative investigation of monitoring data. If the only data you have access to is the metrics you gathered (because you could anticipate they would be useful), then when facing novel problems, your investigations would quickly hit a brick wall. This is why companies are starting to turn to structured events.

Structured events are essentially collections of key-value pairs your service publishes that describe what that service is doing. This may sound familiar: the events from the event-driven architectures you read about in Part 9 are structured events; the structured logs from earlier in this blog post are structured events; and you could even publish all the types of metrics you read about earlier in this blog post as structured events. The idea is to instrument your services to publish structured events that contain all the interesting data about that service and to set up tooling to consume these events, and store them in such a way that you can efficiently analyze what’s happening in your system. What sort of tooling is that?

Note that structured events have many dimensions (many key-value pairs): the time the event happened, the service the event happened in, the upstream service that called this one, the host the service was running on, the ID of the user who was accessing that service, and so on. Moreover, many of these dimensions have a high cardinality, which is a measure of how many distinct values a dimension can have. For example, the user ID dimension could be set to millions of values, depending on how many users you have in your system. So you need tooling that can efficiently store and analyze data with high dimensionality and cardinality.

I haven’t found a widely accepted name for this sort of tooling—I’ve heard "observability platform," "application visibility platform," "software debugging tool," and others—so for the purposes of this blog post series, I’ll use observability tooling specifically to refer to tools for storing and analyzing structured event data with high dimensionality and cardinality. Honeycomb released one of the first publicly available observability tools in 2016 and did a lot of the leg work to popularize the concept of observability. Since then, other observability tools have appeared on the market, such as SigNoz and Uptrace (full list), and many of the monitoring tools you saw earlier, such as New Relic and Datadog, have started to add observability features (or at least, to use "observability" in their marketing).

The ability of observability tools to efficiently slice and dice data with high dimensionality and cardinality is what allows you to iteratively interrogate your systems without knowing in advance the questions you’ll need to ask. For example, let’s say latency is spiking on your website, and you don’t know why. The following walkthrough of a sample debugging session shows how you might debug this issue by using an observability tool:

While you can set up metrics and alerts ahead of time for something like latency, there’s no way to know ahead of time all the queries you would need to do in the debugging session. And even if after this incident you added metrics and dashboards for the queries that helped resolve that incident, the next issue you have to debug will likely require totally different queries. This sort of iterative search process is where observability tools shine, as they enable you to find answers when you don’t know, and can’t know, the questions you’ll have to ask ahead of time.

Another class of tools that falls under the observability umbrella are tracing tools, discussed next.

When you have an architecture with many services, a single request from a user may result in dozens of internal requests within your architecture. For example, when your frontend service receives a request, it may send requests to services A, B, and C; service A, in turn, may send requests to services D, E, and F, while service B sends requests to services G, H, and I; and so on, as shown in Figure 98.

This diagram is static, but in the real world, your services and request flows are going to be changing all the time. Beyond a certain scale, understanding the request flow and debugging problems within it can become a serious challenge. Therefore, many companies turn to distributed tracing, a method of tracking requests as they flow through a distributed system.

The basic idea is that when you get the request that initiates the flow, such as a user’s request to a frontend, you instrument your code to assign a unique ID to that request, called a trace ID, and you propagate that trace ID throughout all the downstream requests. Each of your services can then publish structured events that include the trace ID, along with other useful data such as a timestamp and how long the request took to process, and a distributed tracing tool can stitch all these events back together. That way, for every request, you can get a waterfall diagram similar to Figure 99.

Waterfall diagrams consist of a series of nested spans, one for each request, which show you which services were involved, the order of requests (which reveals dependencies between services), how long the requests took, and other useful metadata. These sorts of diagrams help with debugging problems and performance tuning in a microservices architecture.

Numerous tools are available for instrumenting your code and collecting and analyzing distributed traces. Several of these tools focus primarily on distributed tracing, including Zipkin and Jaeger (full list). Many of the observability tools I mentioned earlier (e.g., Honeycomb, SigNoz, Uptrace) also have first-class support for tracing. Finally, many existing monitoring tools added varying levels of support for distributed tracing as well (e.g., Grafana, Dynatrace, New Relic, and Datadog).

These days, using OTel is often the best approach. It’s open source; works with many programming languages, orchestration tools, and metrics backends; and it’s a single toolkit you can use to collect all your telemetry data (metrics, logs, and traces). Also, as you saw in Part 7, some service mesh tools provide distributed tracing out of the box, such as Istio, which uses OTel under the hood.

You’ve now seen multiple tools for understanding what’s happening in production. If you get good enough with these tools, that opens the door to an approach known as testing in production.

One way to think of monitoring is as the yin to the automated testing yang. That is, you use automated tests to catch some types of errors, and you use monitoring to catch other types. Automated testing is great for catching errors when (a) you can anticipate the error and (b) you can simulate production sufficiently to replicate the error. As you saw in Section 10.3.1, as your company scales, you’ll hit more and more errors that you can’t anticipate. Similarly, as your company scales, you’ll find it harder and harder to simulate production. If you ever get to the scale of a Google or Meta, with billions of users and petabytes of data, there’s just no practical way to have a faithful copy of production.

As a result, some larger companies don’t bother—that is, they don’t have staging environments. Instead, they perform testing in production (TIP)—testing software by deploying it to production and seeing what happens. This may sound reckless and dangerous, and if not done correctly, it can be. However, the reality is that we all test in production to some extent; every deployment, every configuration change, every single user request is a test. It might work, or it might hit a bug. The idea with TIP is that, at a certain scale, the effort to do testing in preproduction environments becomes so high that you may actually be better off investing that effort into making your production systems more resilient. Instead of spending thousands of hours trying to catch every bug with automated tests, you spend some of that time reducing the impact of bugs.

This doesn’t mean you throw all automated testing out the window and just thoughtlessly start tossing untested code into production while screaming Leeeeeeerooooooy Jenkins. If anything, TIP requires more discipline to do well, including the following:

If you do TIP correctly, you’ll see significant improvements in all your DORA metrics. Deployment frequency will go up, and lead times will go down, as deploying will be something every team does dozens of times per day as part of its normal coding process, rather than only at the end. Also, change failure rates and recovery time will go down, as your tooling will be designed to limit the blast radius and quickly recover from errors.

That said, testing in production is not without risks and challenges. For each feature you work on, you will constantly need to balance the amount of time you invest in testing before production versus the amount of testing you want to do in production. When the cost of bugs is high, you’ll want to lean more heavily toward automated testing before production, such as for the following types of features:

For these sorts of features, do as much testing before production as you possibly can. On the other hand, when the cost of bugs is low, you’ll want to lean more heavily toward testing in production. For example, if an employee at Meta is testing in production, and that leads to a bug that breaks the Facebook news feed for a small number of users, it’s just not that big of a deal. For this very reason, Meta’s motto used to be "move fast and break things." The company eventually changed this motto to "move fast with stable infra," which in some ways, does an even better job of encapsulating the TIP philosophy of speed and resiliency.

That said, things will go wrong even with the most resilient systems, and when they do, you’ll want to be alerted about it. This is the focus of the next section.

For each alert, you define triggers, or rules, for when a notification should be sent to someone. You typically define triggers in terms of various metrics, events, or logs exceeding one of the following types of thresholds:

Once you’ve defined a bunch of triggers, the next question is what to do when a trigger goes off. Whenever possible, the default action should be to deal with it by using automation, such as auto healing (e.g., replace a server or container) or auto scaling (e.g., scale the number of servers or containers up or down). If you can’t automatically handle a trigger, the next step is to notify a human, as discussed next.

Although this entire section is about alerts, the reality is that you should use alerts sparingly. I’m specifically referring to the kinds of alerts that notify a person, interrupting whatever they are doing, such as an alert on a pager or, as is more common these days, a smartphone. Alerts of this kind have a significant cost. First, they interrupt work. At some companies, alerts go off all the time, so the Operations team never has time for anything other than "firefighting." Second, they interrupt personal life. Since alerts may come up at any time—in the middle of dinner, in the middle of the night, or during your Christmas holiday—they can significantly interfere with your life outside of work.

If alerts go off too often, your team members will experience alert fatigue, becoming stressed, unproductive, and unhappy. In the best case, they’ll start to ignore alerts, as in "The Boy Who Cried Wolf" parable. In the worst case, your team members will burn out and leave. To avoid alert fatigue, you should use an alert only if the trigger meets all the following criteria:

If a trigger doesn’t meet all three criteria, then instead of an alert, you should send a nonurgent notification, such as filing a ticket in an issue-tracking system or sending a message to a chat room (e.g., Slack or Microsoft Teams) or mailing list that your Operations team checks from time to time. That way, you know someone will see and investigate the issue, without forcing them to drop everything to do it.

If a trigger does meet all these criteria, you send an alert, typically to the team members who are on call, as per the next section.

If you’re at a tiny startup and an incident occurs, you might alert everyone. As your company grows, this becomes inefficient and counterproductive, so having an on-call rotation is more common: you create a schedule that assigns specific team members to be responsible for responding to alerts at specific times. In the past, you might have managed the on-call rotation with a spreadsheet and had the on-call team members carry pagers, but these days, it’s more common to use dedicated software to manage the rotation and send alerts to smartphones. Some of the major players in this space are PagerDuty and Opsgenie (full list).

If you’re on call and an alert goes off, you’ll need to resolve the incident by going through an incident-response process.

Whenever an incident occurs, your team should follow an incident-response plan, which details the steps to resolve that issue. In small startups, this is often an ad hoc process, but as your company grows, you’ll want to get the plan down in writing. In fact, this is a requirement of many compliance standards (e.g., SOC 2). This gives your on-call teams a systematic, repeatable way to resolve incidents, which should give them more confidence and allow them to resolve issues faster (as compared to, say, randomly pushing buttons and hoping it works). Every incident-response plan is different, but most plans include the following items:

Now that you’ve seen the basics of alerts, let’s try an example using CloudWatch.

Earlier in this blog post, you created an asg-sample module that deployed an ASG with an ALB, Route 53 health checks, and a CloudWatch dashboard with some metrics. Let’s set up an alert for one of those metrics—the Route 53 health check—by using CloudWatch alarms. You’ll want the trigger to go off when the HealthCheckStatus metric drops below 1, as that indicates the website is down. When the trigger goes off, it will publish a message to Amazon Simple Notification Service (SNS), and you can subscribe to SNS to get notifications via email or SMS.

Add a CloudWatch alarm to main.tf in the asg-sample module, as shown in Example 193.

Here’s what this code does:

To make it easier for you to test out this alarm, you can also update the asg-sample module to automatically subscribe you to the SNS topic, as shown in Example 194.

Run apply to deploy these changes:

After a minute or two, you’ll get an email confirming your SNS subscription. Make sure to click the link in the Subscribe URL. Now that you’re subscribed and the alarm is deployed, let’s see if it works. To do that, you need to do something fun: break your own sample app! You can get creative with the way you do this. One option is to go into the AWS Load Balancers Console, select the sample-app ALB, and click Actions → "Delete load balancer." After a few minutes, the Route 53 health checks should start failing, and you should get an email similar to Figure 100.

If you see an email with subject "ALARM: sample-app-is-down," then congrats, alerts are working! You now have a way to monitor your software 24-7, and to be notified as soon as anything goes wrong.

If you want to fix the alert you’re seeing, you can run apply one more time to redeploy whatever infrastructure you destroyed. If you’re done testing, commit your latest code to Git, and then run destroy to clean everything up.