Analyze an existing product experiment

Why this matters

Experiment analysis is useful only when assignment, exposure, and outcomes are measured consistently. A variant label on a conversion event is not enough: it can exclude exposed users who never converted and make every variant look better than it was.

Agentry can verify the signals and analyze an experiment your application already runs. It is not a production feature-flag runtime. Owner-authenticated feature management endpoints must not be proxied through an app server or embedded in deployed code.

What you get

• An answerability check for exposure, outcome, identity, and required properties
• Exposure and outcome counts for exact, disclosed time bounds
• Conversion or revenue-per-assignment comparisons with sample caveats
• A decision memo that separates observations, hypotheses, and missing proof

Walk through it

GET /v1/projects/:project_id/signal-map
GET /v1/projects/:project_id/onboarding
# Require status: "verified"; use live event/property reads for the selected signals.
GET /v1/projects/:project_id/event-names
GET /v1/projects/:project_id/event-property-keys?events=experiment_exposure,conversion_completed

The agent should stop if either event is absent, the variant property is missing, identities cannot be joined, or the window contains no complete traffic for one variant. It should name the missing telemetry rather than substituting a plausible event.

For a custom experiment analysis, fetch the current request schema and run a bounded owner-side aggregate query:

Endpoint: POST /v1/projects/:project_id/analytics/query
Exact shape: /v1/openapi.json?path=%2Fv1%2Fprojects%2F%7Bproject_id%7D%2Fanalytics%2Fquery&method=post
Query plan:
  - bind one exact half-open exposure window
  - assign each stable unit to its first valid exposure
  - count every exposed unit, including non-converters
  - count outcomes only after exposure and within the declared horizon
  - aggregate by variant; return no raw identities

{
  "rows": [
    { "variant": "control", "exposures": 1840, "outcomes": 91, "conversion_pct": 4.95 },
    { "variant": "candidate", "exposures": 1812, "outcomes": 97, "conversion_pct": 5.35 }
  ]
}

The rows above illustrate an output shape, not a built-in result or proof that the difference is statistically or commercially meaningful.

The output

Experiment review — <experiment id> — <exact UTC bounds>

Readiness
- exposure and outcome mappings: verified | blocked
- assignment unit and join key: <definition>
- sample-ratio and overlap checks: <result>

Observed
- control: <exposures>, <outcomes>, <rate>
- candidate: <exposures>, <outcomes>, <rate>
- absolute and relative difference: <values>
- revenue and guardrail movement: <values or not measured>

Interpretation
- <evidence-backed statement>
- <hypothesis, explicitly labelled>

Decision for human review
- continue | stop | ship | inconclusive
- why, plus the exact missing proof if inconclusive

Do not claim causality from an observational cohort, treat repeated peeking as a fixed-horizon test, or optimize conversion while ignoring revenue, refunds, errors, and other declared guardrails.

Setting it up

Emit one exposure event at the point where the product actually applies the variant, using a server-ingest or browser-public credential appropriate to that execution context. Carry the stable assignment identifier, experiment id, variant, and assignment version. Emit the business outcome from the trusted system that confirms it, using the same stable identity.

Keep runtime assignment in your existing scoped flag provider or app-owned deterministic allocator. Agentry’s private owner key is for interactive reads and management, never deployed evaluation.

Variations

• “Compare revenue per exposed account, not only checkout conversion.”
• “Check whether the result changes by device, plan, or acquisition source, but label this exploratory.”
• “Review error and refund guardrails before recommending a rollout.”
• “Tell me exactly why this experiment is not yet decision-ready.”