A/B the free-shipping threshold

Why this matters

The free-shipping threshold is the most under-tested lever in ecommerce. Most shops pick a number — $50 because the founder thought it sounded nice — and never revisit it. Meanwhile:

• Too low: you’re eating shipping costs on small orders, AOV stays flat
• Too high: shoppers hit the cart, see the shipping fee, bail
• “Just right”: shoppers add one extra item to clear the threshold, AOV jumps 8-15%

The trick is the AOV/conversion tradeoff. Raising the threshold should drop conversion slightly but raise AOV more — net revenue up. Lowering it should boost conversion but flatten AOV. You can’t tell where you are on the curve without a test, and the only winning move is to test it.

What you get

• A flag-controlled, stable 50/50 split per distinct_id
• Two published dashboards: AOV-by-variant and order-volume-by-variant
• After 14 days, a clear “ship $50” / “ship $75” / “no signal — split test was a wash” verdict

Walk through it

agentry_create_feature_flag {
  project_id: "default",
  key: "free_shipping_threshold",
  name: "Free shipping threshold A/B",
  variants: [
    { key: "fifty",      rollout_percentage: 50, payload: { threshold: 50 } },
    { key: "seventy_five", rollout_percentage: 50, payload: { threshold: 75 } }
  ]
}

{
  "flag_id": "flag_ship_1",
  "key": "free_shipping_threshold",
  "variants": [
    { "key": "fifty", "rollout_percentage": 50 },
    { "key": "seventy_five", "rollout_percentage": 50 }
  ]
}

agentry_publish_query {
  recipe_id: "ab_test_compare",
  description: "Free-shipping A/B — AOV by variant",
  params: {
    flag_key: "free_shipping_threshold",
    metric: "aov",
    revenue_property: "total",
    conversion_event: "order_completed",
    window_days: 14
  }
}

{ "publication_id": "pub_ship_aov", "public_url": "https://agentry.sh/q/pub_ship_aov?key=agp_…" }

agentry_publish_query {
  recipe_id: "ab_test_compare",
  description: "Free-shipping A/B — orders per visitor",
  params: {
    flag_key: "free_shipping_threshold",
    metric: "conversion_rate",
    conversion_event: "order_completed",
    window_days: 14
  }
}

{ "publication_id": "pub_ship_conv", "public_url": "https://agentry.sh/q/pub_ship_conv?key=agp_…" }

const { payload } = await evaluateFlag("free_shipping_threshold", distinct_id);
const remaining = Math.max(0, payload.threshold - cart.subtotal);
showShippingMessage(remaining); // "Add $12 more for free shipping"

Critical to also include free_shipping_threshold_variant as a property on order_completed — that’s what the dashboards group by. I’ll check back in 14 days.

The output (14 days later)

Free-shipping A/B (14d)

Variant        Visitors    Orders     Conv %      AOV       Revenue/Visitor
─────────────────────────────────────────────────────────────────────────────
fifty           48,210     2,941     6.10%      $58.30     $3.56
seventy_five    47,981     2,808     5.85%      $71.20     $4.16

Conversion: -4.1% (seventy_five)
AOV:        +22.1% (seventy_five) ★
Rev/Visit:  +16.9% (seventy_five) ★★

Statistical confidence: 99.1% (p=0.009)

Verdict: SHIP $75.
  Conversion drops slightly (people who would have spent <$75 still buy
  — they just add an item to clear the threshold). AOV jumps $13 per
  order. Net revenue up 16.9% per visitor.

At your annual volume that's ~$2.1M of additional revenue if you ship it.

Suggested follow-up:
  - Re-run with $75 vs $100 — is there even more room?
  - Segment results by category (heavy items vs light) — shipping
    economics differ
  - Test "free shipping at $50, OR $5 shipping" — two-tier pricing

Setting it up

1. Read the flag at the cart display. The threshold drives the “add $X for free shipping” hint and the actual shipping calculation:

// Server-side eval — keeps the api_key off the client.
async function evaluateFlag(key: string, distinct_id: string) {
  const res = await fetch(
    `https://api.agentry.sh/v1/projects/${PROJECT_ID}/feature-flags/evaluate`,
    {
      method: "POST",
      headers: {
        "Authorization": `Bearer ${process.env.AGENTRY_API_KEY}`,  // agk_… (NOT the DSN)
        "Content-Type": "application/json",
        "User-Agent": "myshop/1.0",  // REQUIRED — Cloudflare 403s default UAs
      },
      body: JSON.stringify({ key, distinct_id }),
    },
  );
  return res.json();  // { value: "fifty" | "seventy_five" | … }
}

2. Tag order_completed with the variant so the dashboard knows which arm an order came from:

const variant = await evaluateFlag("free_shipping_threshold", user.email);

await fetch(`https://api.agentry.sh/v1/analytics/${PROJECT_ID}/`, {
  method: "POST",
  headers: { /* same */ },
  body: JSON.stringify({
    event: "order_completed",
    distinct_id: user.email,
    properties: {
      total: order.total,
      item_count: order.items.length,
      free_shipping_threshold_variant: variant.variant,
    },
  }),
});

3. Don’t mix in promotions during the test. A site-wide 20% off coupon will dominate the shipping signal — pause concurrent promos for the test window.

Variations

• “Three-arm: $50, $75, $100. Find the AOV maximum.”
• “$75 free shipping vs flat $5 shipping. Sometimes flat-fee beats free with a threshold.”
• “Per-category test — apparel might want $75, heavy goods might want $150 because shipping costs more.”
• “Show me the cart-value distribution in each arm — are people clustering right at the threshold?”