Compare error rates between staging and prod

Why this matters

Staging exists to catch bugs before they hit prod. In practice, nobody looks at staging error rates — staging is “where my own stuff runs,” not “the canary.”

Sentry treats staging and prod as separate projects, which means side-by-side comparison is a Sunday-night SQL session. Datadog can chart both, but the question “which fingerprints exist in staging that don’t exist in prod yet?” needs a join nobody writes.

Agentry stores both environments in the same project with environment as a tag. One prompt diffs the fingerprint sets and tells you the new bugs riding on the next train to prod. Catch them on Friday afternoon, ship the fix in the same deploy.

What you get

• A list of fingerprints present in staging only (= queued for prod)
• A list of fingerprints 10x worse in staging than prod (= getting worse on the new branch)
• For each, a one-line interpretation + which deploy introduced it

Walk through it

Endpoint: GET /v1/projects/:project_id/cases
Exact shape: /v1/openapi.json?path=%2Fv1%2Fprojects%2F%7Bproject_id%7D%2Fcases&method=get
Use query parameters, not a JSON body.
Query/filter concept:
  status: "open",
  since: "24h",
  group_by: "environment"

{
  "by_environment": {
    "production": [
      { "fingerprint": "f_72ac", "event_count": 412, "users_affected": 88 },
      { "fingerprint": "f_91dd", "event_count": 89,  "users_affected": 14 }
    ],
    "staging": [
      { "fingerprint": "f_72ac", "event_count": 18,  "users_affected": 3 },
      { "fingerprint": "f_91dd", "event_count": 4,   "users_affected": 2 },
      { "fingerprint": "f_b32e", "event_count": 71,  "users_affected": 6 },
      { "fingerprint": "f_5c10", "event_count": 22,  "users_affected": 4 }
    ]
  }
}

Endpoint: GET /v1/cases/:case_id
Exact shape: /v1/openapi.json?path=%2Fv1%2Fcases%2F%7Bcase_id%7D&method=get
Use query parameters, not a JSON body.

{
  "id": "f_b32e",
  "error_type": "TypeError",
  "message": "Cannot read properties of null (reading 'workspaceId')",
  "stack_top": "src/api/billing/seats.ts:74",
  "environment": "staging",
  "first_seen_at": 1747280000,
  "deploy_sha_at_first_seen": "9c3a210"
}

The output

Environment diff (staging vs production, last 24h)

In STAGING only (= riding the next deploy to prod):

   71 events / 6 users    TypeError: Cannot read properties of null
                          src/api/billing/seats.ts:74 (workspace.workspaceId)
                          ↗ introduced by staging deploy 9c3a210 — 14h ago

   22 events / 4 users    500 /api/integrations/slack
                          Missing SLACK_SIGNING_SECRET in staging env
                          ↗ probably env-only; unlikely to repro in prod

Worse in STAGING than PROD (10x+):
   (none this window)

In PROD only (= broken right now, fix separately):

  412 events / 88 users   TypeError: Cannot read properties of undefined ('id')
                          src/middleware/auth.ts:42
                          ↗ existing case, see f_72ac

Setting it up

This only works if both environments send errors to the same project and tag the environment field. The hard split (separate Sentry projects per env) is the wrong design for diffing — keep them together.

// Agentry.ts — the small fetch helper your agent generates at install time.
const SERVER_KEY = process.env.AGENTRY_SERVER_API_KEY!;

export async function logError(err: Error, ctx?: Record<string, unknown>) {
  await fetch("https://api.agentry.sh/v1/logs/", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${SERVER_KEY}`,
      "Content-Type": "application/json",
      "User-Agent": "myapp/1.0",  // REQUIRED — Cloudflare 403s default UAs
    },
    body: JSON.stringify({
      message: err.message,
      stack: err.stack,
      environment: process.env.NODE_ENV ?? "production",  // ← THIS LINE
      release: process.env.AGENTRY_DEPLOY_SHA,
      ...ctx,
    }),
  });
}

Create separate server_ingest credentials for staging and production, bind both to the same project, and store each as AGENTRY_SERVER_API_KEY only in its own environment. The environment field drives the comparison while separate credentials preserve independent revocation and provenance.

Variations

• “Same diff but for the last 7 days — anything chronic in staging that we keep forgetting to fix before promoting?”
• “Compare staging vs prod error rates by endpoint — which routes are flakier in staging?”
• “Show fingerprints that appear in staging, then 1-2 deploys later appear in prod. What’s our staging-as-canary precision?”
• “On demand, summarize staging-only fingerprints for an engineer to review and optionally share with #eng-staging.”