Compare error rates between staging and prod

Why this matters

Staging exists to catch bugs before they hit prod. In practice, nobody looks at staging error rates — staging is “where my own stuff runs,” not “the canary.”

Sentry treats staging and prod as separate projects, which means side-by-side comparison is a Sunday-night SQL session. Datadog can chart both, but the question “which fingerprints exist in staging that don’t exist in prod yet?” needs a join nobody writes.

Agentry stores both environments in the same project with environment as a tag. One prompt diffs the fingerprint sets and tells you the new bugs riding on the next train to prod. Catch them on Friday afternoon, ship the fix in the same deploy.

What you get

• A list of fingerprints present in staging only (= queued for prod)
• A list of fingerprints 10x worse in staging than prod (= getting worse on the new branch)
• For each, a one-line interpretation + which deploy introduced it

Walk through it

agentry_list_cases {
  project_id: "default",
  status: "open",
  since: "24h",
  group_by: "environment"
}

{
  "by_environment": {
    "production": [
      { "fingerprint": "f_72ac", "event_count": 412, "users_affected": 88 },
      { "fingerprint": "f_91dd", "event_count": 89,  "users_affected": 14 }
    ],
    "staging": [
      { "fingerprint": "f_72ac", "event_count": 18,  "users_affected": 3 },
      { "fingerprint": "f_91dd", "event_count": 4,   "users_affected": 2 },
      { "fingerprint": "f_b32e", "event_count": 71,  "users_affected": 6 },
      { "fingerprint": "f_5c10", "event_count": 22,  "users_affected": 4 }
    ]
  }
}

agentry_get_case { case_id: "f_b32e" }

{
  "id": "f_b32e",
  "error_type": "TypeError",
  "message": "Cannot read properties of null (reading 'workspaceId')",
  "stack_top": "src/api/billing/seats.ts:74",
  "environment": "staging",
  "first_seen_at": 1747280000,
  "deploy_sha_at_first_seen": "9c3a210"
}

The output

Environment diff (staging vs production, last 24h)

In STAGING only (= riding the next deploy to prod):

   71 events / 6 users    TypeError: Cannot read properties of null
                          src/api/billing/seats.ts:74 (workspace.workspaceId)
                          ↗ introduced by staging deploy 9c3a210 — 14h ago

   22 events / 4 users    500 /api/integrations/slack
                          Missing SLACK_SIGNING_SECRET in staging env
                          ↗ probably env-only; unlikely to repro in prod

Worse in STAGING than PROD (10x+):
   (none this window)

In PROD only (= broken right now, fix separately):

  412 events / 88 users   TypeError: Cannot read properties of undefined ('id')
                          src/middleware/auth.ts:42
                          ↗ existing case, see f_72ac

Setting it up

This only works if both environments send errors to the same project and tag the environment field. The hard split (separate Sentry projects per env) is the wrong design for diffing — keep them together.

// Agentry.ts — the small fetch helper your agent generates at install time.
const DSN = process.env.AGENTRY_DSN!;
const PROJECT_ID = process.env.AGENTRY_PROJECT_ID!;

export async function logError(err: Error, ctx?: Record<string, unknown>) {
  await fetch(`https://api.agentry.sh/v1/logs/${PROJECT_ID}/`, {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${DSN}`,
      "Content-Type": "application/json",
      "User-Agent": "myapp/1.0",  // REQUIRED — Cloudflare 403s default UAs
    },
    body: JSON.stringify({
      message: err.message,
      stack: err.stack,
      environment: process.env.NODE_ENV ?? "production",  // ← THIS LINE
      release: process.env.GIT_SHA,
      ...ctx,
    }),
  });
}

Use the same AGENTRY_DSN (or a per-env DSN that points at the same project) in staging and prod. The environment field is what drives the split.

Variations

• “Same diff but for the last 7 days — anything chronic in staging that we keep forgetting to fix before promoting?”
• “Compare staging vs prod error rates by endpoint — which routes are flakier in staging?”
• “Show fingerprints that appear in staging, then 1-2 deploys later appear in prod. What’s our staging-as-canary precision?”
• “Daily cron: post a summary of staging-only fingerprints to #eng-staging.” (uses a Routine — see staging-watcher)