Test a post-purchase upsell screen

Why this matters

The post-purchase window is the most overlooked piece of real estate in ecommerce. The customer:

• Just typed their card and clicked “Buy”
• Has card-on-file authorized
• Is in a buying mood
• Trusts you (they just paid you)

Conversion on a one-click upsell at this moment is typically 3-5× higher than normal browsing. Most shops still show a vanilla confirmation page — “Thank you for your order, check your email.” Money on the table.

The classic experiment: half your buyers see a relevant one-click add (“Want the matching belt for $19 — one click, ships in the same box”), half see the normal page. Measure repeat-purchase rate. Agentry runs the A/B test with one flag and one published query.

What you get

• A feature flag that splits buyers 50/50 between “see upsell” and “control”
• A query that compares repeat-purchase rate (or revenue) between the cohorts
• A clear “ship the upsell” or “don’t — the offer wasn’t strong enough” verdict

Walk through it

agentry_create_feature_flag {
  project_id: "default",
  key: "show_post_purchase_upsell",
  name: "Post-purchase upsell screen A/B",
  variants: [
    { key: "control", rollout_percentage: 50, payload: { show_upsell: false } },
    { key: "treatment", rollout_percentage: 50, payload: { show_upsell: true } }
  ]
}

{
  "flag_id": "flag_upsell_1",
  "key": "show_post_purchase_upsell",
  "variants": [
    { "key": "control", "rollout_percentage": 50 },
    { "key": "treatment", "rollout_percentage": 50 }
  ]
}

agentry_analytics_query {
  project_id: "default",
  query: "
    with first_orders as (
      select
        distinct_id,
        properties.show_post_purchase_upsell_variant as variant,
        min(timestamp) as first_order_ts
      from events
      where event = 'order_completed'
        and timestamp > now() - interval 44 day
        and properties.show_post_purchase_upsell_variant in ('control', 'treatment')
      group by distinct_id, variant
    ),
    repeats as (
      select fo.variant, fo.distinct_id,
        countIf(e.timestamp > fo.first_order_ts
                and e.timestamp < fo.first_order_ts + interval 30 day) as repeats_30d
      from first_orders fo
      left join events e on e.distinct_id = fo.distinct_id and e.event = 'order_completed'
      group by fo.variant, fo.distinct_id
    )
    select
      variant,
      count() as buyers,
      countIf(repeats_30d > 0) as repeat_buyers,
      countIf(repeats_30d > 0) / count() as repeat_rate
    from repeats
    group by variant
  "
}

{
  "rows": [
    { "variant": "control",   "buyers": 4210, "repeat_buyers":  301, "repeat_rate": 0.0715 },
    { "variant": "treatment", "buyers": 4198, "repeat_buyers":  512, "repeat_rate": 0.1220 }
  ]
}

agentry_analytics_query {
  project_id: "default",
  query: "
    -- same shape, but only count repeats that happen 24h+ after first order
    select variant,
      count() as buyers,
      countIf(repeats_after_24h > 0) as true_returners,
      countIf(repeats_after_24h > 0) / count() as true_repeat_rate
    from (
      select fo.variant, fo.distinct_id,
        countIf(e.timestamp > fo.first_order_ts + interval 24 hour
                and e.timestamp < fo.first_order_ts + interval 30 day) as repeats_after_24h
      from first_orders fo
      left join events e on e.distinct_id = fo.distinct_id and e.event = 'order_completed'
      group by fo.variant, fo.distinct_id
    )
    group by variant
  "
}

{
  "rows": [
    { "variant": "control",   "buyers": 4210, "true_returners": 280, "true_repeat_rate": 0.0665 },
    { "variant": "treatment", "buyers": 4198, "true_returners": 310, "true_repeat_rate": 0.0738 }
  ]
}

The output

Post-purchase upsell A/B (14d)

Variant       Buyers   Same-session   24h+ repeats   Combined
                       upsell %       (true repeat)  30d repeat
─────────────────────────────────────────────────────────────────
control        4,210      n/a              6.65%       7.15%
treatment      4,198      4.95%            7.38%      12.20%

Same-session lift:  +4.95 pp ★★★ (~$8k extra revenue per cohort)
True repeat lift:   +0.73 pp ★  (real, modest, not cannibalization)
Combined 30d lift:  +5.05 pp ★★

Verdict: SHIP IT. Flip to 100% treatment.

Suggested follow-ups:
  - Try variant copy: "Add for $19" vs "Bundle and save $5"
  - Test offer relevance: matched-by-AI vs always-the-belt
  - Measure refund rate per arm — upsold items sometimes refund higher
  - Add an A/B/C test for the upsell discount (0%, 10%, 20%)

Setting it up

1. Read the flag on the order-confirmation page to decide whether to render the upsell:

// app/checkout/[order]/page.tsx
const { variant, payload } = await evaluateFlag(
  "show_post_purchase_upsell",
  user.email,
);

return (
  <>
    <OrderSummary order={order} />
    {payload?.show_upsell && <UpsellPanel order={order} />}
  </>
);

2. Tag order_completed with the variant so the comparison query can group:

await fetch(`https://api.agentry.sh/v1/analytics/${PROJECT_ID}/`, {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.AGENTRY_DSN}`,
    "Content-Type": "application/json",
    "User-Agent": "myshop/1.0",  // REQUIRED — Cloudflare 403s default UAs
  },
  body: JSON.stringify({
    event: "order_completed",
    distinct_id: user.email,
    properties: {
      order_id: order.id,
      total: order.total,
      show_post_purchase_upsell_variant: variant.variant,
      is_upsell: order.parent_order_id ? true : false,
    },
  }),
});

3. Use a stable distinct_id. First-time guest checkouts should fall back to a visitor cookie so the same person sees the same variant if they come back.

Variations

• “Three arms: no offer, 10% off offer, ‘add-on for $X’ offer. Which framing wins?”
• “Segment results by first-order AOV — does the upsell work better on big-cart customers or small-cart?”
• “Test the offer item — same upsell layout, different recommended product (rule-based vs AI-recommended).”
• “Add a survey to the upsell screen — for people who dismiss it, ask why.”