PagerDuty Integration

PagerDuty Integration

Sentinel opens a PagerDuty incident when a monitor goes down and resolves it when the monitor recovers, so PagerDuty's escalation policies, schedules and phone calls sit downstream of Sentinel's multi-region checks. It uses PagerDuty's Events API v2, which needs nothing more than an integration key from the service you want paged.

Setup

1. In PagerDuty, open Services, pick the service that should receive Sentinel incidents (or create one), and open its Integrations tab.
2. Add an integration of type Events API V2 and copy its Integration Key (32 hex characters).
3. In Sentinel, go to your team's Integrations page and click Add service under PagerDuty.
4. Name the service, paste the key, choose which severities page it, and save.
5. Click Test. Sentinel opens a test incident on the service and resolves it straight away, so you see a resolved incident in PagerDuty's timeline and a confirmation in Sentinel. A rejected key is reported on the spot.

A team can connect several services, for example one per client or one per escalation policy. Each has its own key and severity choices. Keys are encrypted at rest and never shown again after saving.

What opens and resolves an incident

Only transitions with a matching recovery reach PagerDuty, so nothing opens an incident that Sentinel could never close:

Severity and urgency are different things in PagerDuty. A new service pages at high urgency for every incident whatever the severity; to have warnings arrive as low urgency, set the service's incident urgency to "dynamic" (based on severity) in its settings.

Expiring certificates, expiring domains, slow responses and DNS changes are not sent to PagerDuty. They have no recovery event, so an incident opened for one would stay open forever. Route those to email, Slack or a webhook instead.

Deduplication

Every event carries a dedup key made of the monitor and the failure family, for example sentinel-monitor-123-availability. Repeated failures while an incident is open append to it rather than paging again, the recovery resolves exactly the incident its outage opened, and a keyword failure on a monitor that is also down stays a separate incident. Once an incident is resolved, the next outage opens a new one.

Which severities page

Each service has its own severity choices (outages, warnings, recoveries). The default is outages only. Recoveries are special: a recovery always resolves the incident its outage opened, whatever the service is ticked for, so an on-call responder never has to close a Sentinel incident by hand.

A monitor's own Notifications page has a PagerDuty row as well. Untick a severity there to keep that monitor off PagerDuty for it, for example a staging site that should alert Slack but never page anyone. Over the API and the Terraform provider the same routing is the pagerduty key inside notification_settings.channels.

What PagerDuty receives

{
  "event_action": "trigger",
  "dedup_key": "sentinel-monitor-123-availability",
  "client": "Sentinel",
  "client_url": "https://sentinel.rootstuff.io/acme/monitors/123",
  "links": [
    { "href": "https://sentinel.rootstuff.io/acme/monitors/123", "text": "Monitor in Sentinel" },
    { "href": "https://sentinel.rootstuff.io/acme/incidents/456", "text": "Incident in Sentinel" }
  ],
  "payload": {
    "summary": "Storefront: down (Service Unavailable (503) - The server is temporarily unavailable)",
    "source": "shop.example.com",
    "severity": "critical",
    "timestamp": "2026-09-16T12:51:12Z",
    "component": "http",
    "group": "Acme Studio",
    "class": "availability",
    "custom_details": {
      "event": "monitor.down",
      "monitor_id": 123,
      "url": "https://shop.example.com",
      "reason": "Service Unavailable (503) - The server is temporarily unavailable",
      "detected_from": "ash",
      "previous_status": "online",
      "current_status": "offline",
      "incident_id": 456
    }
  }
}

The resolve event carries only event_action and the same dedup_key. Push (heartbeat) monitors never include their ping URL, since it is a secret.

Delivery health

The integrations page shows when each service last delivered, and a running failure streak with PagerDuty's status code when it stops accepting events. Transient failures (connection errors, 5xx, 429) are retried twice within the same alert; a 400 means the key was rejected and is not retried, so fix the key and press Test.

Troubleshooting

• Test says the key was rejected. Copy the key from an Events API V2 integration, not a Generic Webhook, email or REST API key. It is 32 hex characters.
• Nothing paged during an outage. Check the service is not paused, its severities include outages, the monitor's Notifications page has PagerDuty ticked for that severity, and the team owner's plan is Business.
• The incident never resolved. The service was removed or paused between the outage and the recovery. Resolve it in PagerDuty; the next outage opens a fresh incident.