Blog

Error alerts without the noise: group, threshold, snooze

· SyncYak team

  • error-tracking
  • alerting
  • slack

The fastest way to make a team ignore error alerts is to send one Slack message for every exception. A single failing database connection can produce thousands of identical errors in a minute. By the second afternoon, nobody reads the channel, and the alert that mattered goes unseen.

Alert fatigue is a well-documented problem in on-call work: when responders are flooded with low-value notifications, they become desensitized, and the one that gets ignored may be the one that signals a genuine problem. Most of the cure is not a smarter tool. It is a handful of settings and a few habits. This post walks through them, using Cloud Logging as the worked example.

Start with grouping, not alerting

An alert is only as good as the unit it counts. If every occurrence is its own event, you cannot tell ten problems from ten thousand repeats.

Cloud Logging groups occurrences into an item when they share a fingerprint: the exception class and its stack frames, or the message text for log lines. Everything is scoped to one environment, so the same bug in production and staging is two items. That matters for alerting, because you usually want to hear about production and merely see staging.

Before you touch any alert setting, open the item list and check that the grouping looks right. If one bug appears as many items, the fingerprint is picking up something that varies, such as an ID inside a message. Fix that in the message you log, not in the alert rules.

Alert on two events only

Cloud Logging posts to Slack for two kinds of event:

  1. A new item. The first occurrence of a fingerprint. This is always posted, even during a cooldown, because a never-seen error is the most useful signal you get.
  2. A spike. An item reaches a set number of occurrences within a time window.

Everything else is deliberately silent. An item that has already alerted does not alert again on every occurrence. Muted and snoozed items never alert, and occurrences below your minimum level are ignored.

The defaults are a reasonable starting point: a minimum level of error, a spike at 25 occurrences within 300 seconds, and a 1,800-second (30-minute) cooldown. Alerting is off until you turn it on and name a Slack incoming webhook.

Set the minimum level on purpose

Levels run from debug through info, warning and error to critical. The minimum level decides which occurrences can alert at all.

  • Production: error, or critical if your app logs errors for expected failures such as a user entering a bad card number.
  • Staging: leave alerts off, or alert on critical only.
  • A new service you do not trust yet: start at warning for a week, look at what it finds, then raise the level.

Settings resolve through a chain: defaults, then account, then project, then environment. A more specific level overrides the one above it, and a zero or empty field inherits. So you can set sensible values on the project and override just one field on the production environment.

{
  "environment": "production",
  "enabled": true,
  "min_level": "error",
  "cooldown_sec": 3600,
  "spike_count": 50,
  "spike_window_sec": 300
}

This sends the production environment's alerts at error and above, requires 50 occurrences within five minutes to count as a spike, and waits an hour before the same item can alert again. The shape follows the alert settings in the Cloud Logging API reference.

Tune the spike threshold to your traffic

A spike of 25 in five minutes means something different for a site with 200 visits a day than for one with 200,000. Look at a week of real data:

  • If your normal noisy item produces 5 occurrences in five minutes, a threshold of 25 is five times normal. That is a reasonable signal.
  • If your busiest healthy item produces 40, a threshold of 25 will fire constantly. Raise it above that, or fix the item.

Repeat this check after each large release. Alert settings that were right six months ago drift as traffic and code change.

Use snooze for known issues, resolve for fixed ones

Two different situations call for two different tools:

  • Resolved means you fixed it. The item leaves the default list. If it happens again, it reopens automatically as a regression, and that reopening is a signal worth having.
  • Muted hides the item and stops alerts permanently. Use it for errors you have decided to live with, such as a bot hitting a URL you will never serve.
  • Snooze silences alerts until a time you choose, without changing status. The item stays in the list and keeps collecting. Use it for a known issue with a fix scheduled for Thursday. Snooze is capped at your retention window and is included from the Pro plan.

The habit that makes this work: every alert gets one of three outcomes. Fix it and resolve it, snooze it with a date, or mute it with a reason. An alert that is just left alone trains everyone to ignore the channel.

Keep secrets out of the alerts

Quiet alerts are no use if they leak credentials into a chat channel. Cloud Logging masks values before anything is written to disk, and the built-in list covers keys such as authorization, api_key and x-api-key. You can add fields at the account, project or environment level. Masking applies to new occurrences only, so set it before you connect a busy app.

A weekly ten-minute review

Alerting settings are not set-and-forget. Once a week, look at the items that alerted:

  1. Which were real? Fix and resolve them.
  2. Which were known? Snooze with a date, or mute.
  3. Which alerted too often? Raise the spike count or cooldown, or fix the fingerprint.
  4. Did anything real fail to alert? Lower the minimum level for that environment.

How SyncYak helps

Cloud Logging accepts the Rollbar SDKs you may already run: change the endpoint and keep your error handling. Errors are grouped into items, and alerts go to Slack with the new-item and spike rules above. If you run Laravel, Laravel error tracking takes two lines in .env. The Cloud Logging help page covers projects, environments and alert destinations.

The Free plan collects 10,000 events a month with 14-day retention, one project and one Slack destination. See pricing for the paid tiers, including on-demand events past your plan up to a cap you set.

Create a free account, point one app at it, and set the alert level before you invite anyone to the channel.

Start free with SyncYak · All posts