Alerts

Alerts notify you when conditions in your application require attention: errors, latency spikes, metric thresholds, or missing data. Configure channels and rules to get notified through email, Slack, webhooks, GitHub Issues, Pushover, or Telegram, or to page whoever is on call.

How Alerts Work

The alerting system has two parts:

• Channels define where notifications are delivered (email, Slack, webhook, GitHub, Pushover, Telegram, or an escalation policy)
• Rules define what conditions trigger a notification

When a rule's condition is met, TracePath sends a notification through the rule's attached channel. Each rule is linked to exactly one channel, but a channel can be used by multiple rules.

Channels

A channel represents a notification destination. TracePath supports the following channel types:

Each channel has a Test button that sends a sample notification so you can verify the configuration before attaching rules. Escalation-policy channels are the exception: their Test button opens a real page and notifies the on-call responder, so resolve the test page when you are done.

Setting Up Email

To receive alerts via email, add one or more recipient email addresses. TracePath will send an email to all listed recipients when a rule fires.

Setting Up Slack

To send alerts to a Slack channel, you need an Incoming Webhook URL:

1. Go to api.slack.com/apps (opens in a new tab) and click Create New App
2. Choose From scratch, give it a name (e.g. "TracePath Alerts"), and select your workspace
3. In the app settings, navigate to Incoming Webhooks and toggle it on
4. Click Add New Webhook to Workspace, select the channel you want alerts posted to, and click Allow
5. Copy the Webhook URL. It will look like https://hooks.slack.com/services/T.../B.../xxx

Then configure the Slack channel in TracePath:

Setting Up GitHub

To create GitHub issues from alerts, you need a Personal Access Token with issue permissions:

1. Go to GitHub Settings → Developer settings → Personal access tokens → Fine-grained tokens (opens in a new tab)
2. Click Generate new token
3. Under Resource owner, select the account or organization that owns the target repository. If your organization doesn't appear, an org admin needs to enable fine-grained tokens under Organization Settings → Third-party access → Personal access tokens → Settings
4. Set a descriptive name (e.g. "TracePath Alerts") and choose an expiration
5. Under Repository access, select Only select repositories and choose the repo where issues should be created
6. Under Permissions → Repository permissions, set Issues to Read and write
7. Click Generate token and copy it; you won't be able to see it again

Then configure the GitHub channel in TracePath:

Issues opened by this channel can start an AI coding agent in the repository's own GitHub Actions, which investigates the exception against the TracePath API and opens a pull request or comments its analysis. See Auto-Fix.

Setting Up Telegram

To send alerts to Telegram, you need a Bot Token and a Chat ID:

1. Open Telegram and start a chat with @BotFather (opens in a new tab). Send /newbot and follow the prompts to give your bot a name and username. BotFather will reply with a Bot Token in the form 123456789:ABCdefGhIJKlmNoPQRstuVwxyZ. Copy it.
2. Open a chat with your new bot and send it /start so it is allowed to message you.
3. Create a private channel in Telegram for the alerts.
4. Open the channel's settings → Administrators → Add Admin, search for your bot's username, and add it with permission to post messages.
5. Forward any message from the channel to @userinfobot (opens in a new tab); it will reply with the channel's Chat ID (a negative number, e.g. -1001234567890). Copy it including the minus sign.

Then configure the Telegram channel in TracePath:

Webhook Payload

When a webhook channel fires, TracePath sends a JSON POST request with the following fields:

If you configure an HMAC secret on the channel, each request includes an X-TracePath-Signature header containing sha256=<hex> of the request body, which you can use to verify that the request came from TracePath.

Rules

A rule connects a condition to a channel. Every rule has:

Rules belong to a project and only evaluate that project's data. Frontend projects only support the New Issue rule type; all other types monitor backend telemetry.

Rule Types

New Issue

Fires the first time an error with a given fingerprint is seen. Errors are grouped by a normalized stack trace hash, so the same logical error reported many times produces a single alert rather than one per occurrence. If you archive an issue and it occurs again, that first re-occurrence counts as new and fires the rule.

The notification includes the error type, stack trace, app version, server name, the endpoint or task it occurred on, and a deep link to the issue.

The cooldown applies per error, not per rule: two different new errors arriving a minute apart both alert, but the same error will not alert again until its cooldown expires.

Error Regression

Fires when an error whose issue was archived (resolved) occurs again. Regressions are detected by the same grouping, so the alert means the same failure is back, not that a similar-looking one appeared. Use this to catch fixes that did not stick. The cooldown applies per error. There is no extra configuration.

Error Rate

Fires when the percentage of requests returning a 5xx status across all endpoints exceeds the threshold within the lookback window.

Error Count

Fires when the total number of errors recorded in the lookback window reaches the threshold. Captured messages (log-style events without an exception) are not counted.

Endpoint Error Rate

Like Error Rate, but scoped to a single endpoint.

Endpoint P95 / P99

Fires when the P95 (or P99) response time over the lookback window exceeds the threshold. Two separate rule types so you can alert on each percentile independently.

Apdex Drop

Fires when the Apdex score over the lookback window falls below the threshold. TracePath computes Apdex with fixed bands: requests up to 750 ms are satisfied, up to 1500 ms are tolerating, and anything slower or returning 5xx is bad. The score is (satisfied + tolerating / 2) / total, so 1.0 is perfect and anything below roughly 0.85 means users are noticing.

Throughput Drop

Fires when request volume in the lookback window drops by more than the configured percentage compared to the baseline window that immediately precedes it. The baseline count is normalized to the lookback length, so a 15-minute window is compared against the expected 15-minute share of a 60-minute baseline. If the baseline window has no traffic at all, the rule does not fire.

Impact Score (Critical / High / Medium)

Fires when an endpoint's impact score crosses into the configured level. The impact score is a 0 to 1 rating computed over the trailing 24 hours as the worst of five components: Apdex degradation, server error rate, P99 latency, client error rate, and error volume. Medium starts at 0.25, High at 0.50, and Critical at 0.75. Streaming endpoints are excluded.

These rules are transition-based: they fire once when an endpoint newly enters the level, not continuously while it stays there. The endpoint alerts again only if it drops out of the level and later crosses back in. After a server restart the first evaluation establishes a baseline without firing.

The notification names the endpoint, its score, and the dominant reason (for example a P99 of 9.2 s, or a 12% server error rate).

Task Duration

Fires when the P95 duration of background task executions over the lookback window exceeds the threshold.

Task Failure Rate

Fires when the percentage of task executions that recorded an error exceeds the threshold. An execution counts as failed when at least one exception was captured during it, so one execution with three errors counts once.

Metric Threshold

Fires when an aggregated metric value violates the configured comparison over the lookback window. Works for both system metrics (cpu.used_pcnt, mem.used, ...) and custom metrics. If the window contains no data points the rule does not fire; pair it with a No Data rule if you also want to know when the metric stops reporting.

The aggregation spans every series of the metric name, with no tag filter. For metrics tagged per host, per disk, or per table, an avg can hide one bad series behind healthy ones; prefer max (or min for lower-is-bad metrics) so the worst series drives the alert.

No Data

Fires when no telemetry of the selected kind has been received for the configured number of minutes. Use it to detect a stopped service, a broken deploy, or a misconfigured SDK rather than an application-level problem.

AI Trace Cost

Fires when a single AI trace exceeds a cost threshold, catching runaway agent loops or unexpectedly expensive calls as they are ingested rather than at the end of the month.

The cooldown applies per trace name, so two different expensive workflows alert independently.

AI Conversation Cost

Fires when the cumulative cost of a single AI conversation over the last 24 hours crosses the threshold. Where AI Trace Cost catches one expensive call, this catches sessions that get expensive gradually: a runaway agent loop of cheap calls, or a user hammering an assistant.

The cooldown applies per conversation, so one noisy conversation alerts once per cooldown window while a second conversation crossing the threshold still alerts immediately.

AI Flagged Content

Fires when an AI conversation matches a flagged content term. Matching happens at ingest against the language packs selected in the project's AI settings plus your custom terms, so the alert carries the matched terms.

Monitor Down

Fires when a monitor transitions to down, which happens only after its configured number of consecutive failures, and sends a recovery notice when it comes back up. One rule covers every monitor in the project; the alert names the monitor and carries the last probe error.

There are no rule-specific options. Pointing the rule at an escalation channel opens an on-call page instead of sending a message: the page deduplicates per monitor, re-fires never restart its escalation clock, and recovery resolves it automatically.

Evaluation Model and Cost

Rules are evaluated in one of three ways:

• On ingest (New Issue, Error Regression, AI Trace Cost, AI Conversation Cost, AI Flagged Content): evaluated immediately as reports arrive, so alerts fire within seconds of the triggering event.
• On state change (Monitor Down): fires the moment a probe result flips the monitor's up/down state, independent of any poll interval.
• Polled (everything else): a background loop evaluates all enabled polled rules on a 60-second interval, so detection latency for a polled rule is at most one minute.

Each rule type does a different amount of work per evaluation. It does not change your bill — rule evaluation is not metered — but it is worth knowing when a project carries dozens of rules over a lot of data, because the cheapest rule that answers your question is also the one that fires soonest:

Cost-relevant behavior:

• Rules that are snoozed or still in cooldown are skipped entirely, including their queries. Longer cooldowns directly reduce evaluation load.
• The three Impact Score levels share one 24-hour scan per project per poll cycle through a 30-second cache, so adding all three costs the same as adding one.
• On-ingest rules deduplicate per error hash, so a busy error stops generating lookups once it has alerted until its cooldown expires.
• Polled query cost scales with the lookback window. The defaults keep windows short; a 24-hour lookback on a high-traffic project costs proportionally more.

Cooldown and Snooze

Cooldown prevents a rule from firing repeatedly for the same ongoing condition. The default is 15 minutes, configurable per rule.

• For polled rules the cooldown is rule-wide: after a notification is delivered, the rule is not evaluated again until the cooldown expires.
• For on-ingest rules the cooldown is per condition: per error hash for New Issue and Error Regression, per trace name for AI Trace Cost. Different errors can alert back-to-back; the same error cannot.

Rule-level cooldowns are persisted with the notification history, so a platform restart does not cause every rule to re-fire. Per-error cooldown state is held in memory, so an already-alerted error can alert once more after a restart — the failure mode is one duplicate, never a missed alert. Editing a rule resets its cooldown and per-error state so the new configuration takes effect immediately.

Snooze temporarily mutes a rule for a set duration from the dashboard. Snoozed rules are not evaluated at all. Use it during deployments, maintenance windows, or known incidents.

Severity

Each rule has a Severity setting that controls the severity attached to its notifications. The default is Auto, which derives severity from how badly the condition is violated:

You can override the automatic severity by setting it explicitly on a rule to Critical, Warning, or Info. This is useful when you want a normally-warning condition to page as critical, or to downgrade noisy rules to info.

History

All fired notifications are recorded and visible in the History tab on the Alerts page. Each entry shows: