What is considered the best practice when working with alerting notifications?

The best practice in modern monitoring is to create a high-signal, low-noise alerting system. This is achieved by alerting on symptoms that are, or are about to become, externally visible and impact users (e.g., high error rates, increased latency). This approach ensures that every alert is meaningful, actionable, and warrants human intervention. Focusing on symptoms rather than every possible underlying cause prevents "alert fatigue," where operators become desensitized to frequent, low-impact notifications. This philosophy prioritizes system reliability from the end-user's perspective and makes the on-call process more effective and sustainable.

A. Treating all alerts equally leads to alert fatigue, causing critical issues to be overlooked among less important ones.

C. Creating as many alerts as possible generates excessive noise, making it difficult to identify and respond to genuine emergencies.

D. Alerting on every metric is an anti-pattern that produces an unmanageable volume of unactionable alerts, rendering the system useless.

1. Prometheus Official Documentation, Best Practices, "Alerting on what matters": "Your alerting rules should fire on symptoms that are noticeable by the users of your service... Alert on symptoms. Your alerting rules should fire on symptoms that are noticeable by the users of your service... It is better to have a clear dashboard of causes that you can look at when an alert fires, than to receive an alert for a cause that hasn't become a problem yet."

Source: Prometheus Authors. (n.d.). Alerting on what matters. Prometheus Documentation. Retrieved from https://prometheus.io/docs/practices/alerting/

2. Google, "Site Reliability Engineering," Chapter 6: Monitoring Distributed Systems: This foundational text, often used in university curricula, advocates for alerting on symptoms and avoiding alert fatigue. "The rule of thumb for alerting is that the alert’s channel should match the urgency of the alert... Paging a human is a very expensive use of an engineer’s time. If a human is required, the page should be actionable... A simple way to think about whether an alert is good or not is to ask 'Can a human operator take a specific, predefined action in response to this alert?'"

Source: Beyer, B., Jones, C., Petoff, J., & Murphy, N. R. (Eds.). (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly Media, Inc. (Chapter 6, "Monitoring Distributed Systems," Section on "Alerting").

3. University of California, Berkeley, CS 162: Operating Systems and System Programming, Lecture 23: "Distributed Systems Part 2": Course materials discuss the principles of monitoring large-scale systems, referencing Google's SRE model. The lecture notes emphasize the importance of focusing on Service Level Objectives (SLOs) and user-facing symptoms for alerting, stating, "Alert when SLO is in danger of being violated... Don't alert on causes, alert on symptoms."

Source: Culler, D. & Stoica, I. (2022). CS 162: Operating Systems and System Programming, Lecture 23. University of California, Berkeley. Retrieved from https://cs162.eecs.berkeley.edu/static/lectures/23-distsys2.pdf (Slide 59).