DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Newsletter
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Related

  • Demystifying Event Storming: Process Modeling Level Event Storming (Part 2)
  • Importance of VLSI Design Verification and Its Methodologies
  • 5 Vital Steps in Successfully Setting Up Your Startup QA Process
  • 7 Failure Modes Every Production AI Agent Should Be Tested Against

Trending

  • Ingesting Public Data That Fails Silently: Patterns From a TypeScript and Cloudflare Workers Pipeline
  • Choosing a BBD Framework
  • The "Just Read the Code" Myth Is Killing Your AI Workflow
  • Tracing Dangerous Python Code Paths With rce-path

Draft This article hasn't been published, so readers can't see it.

In moderation This article is waiting for review, so readers can't see it yet.

Deleted This article has been deleted, so readers can't see it.

  1. DZone
  2. Culture and Methodologies
  3. Methodologies
  4. A/B Testing Under Shared Constraints: Supply, Inventory, and User Attention

A/B Testing Under Shared Constraints: Supply, Inventory, and User Attention

A practical method for separating incremental system value from cannibalization in marketplace and multi-surface experiments.

By 
Arun Thomas
·
Nov. 16, 26 · Analysis
Post on X (opens in a new tab) Share on LinkedIn (opens in a new tab)
135 Views

Join the DZone community Save articles, join the discussion, and download Refcards and Trend Reports with a free account.

Join for free Already a member? Log in

Why Local Lift Does Not Mean System Growth

A marketplace team tests a new ranking policy. On the home surface, the share of user activity rises from 30% in control to 45% in treatment: a 15-percentage-point lift. Assignment is balanced, tracking is complete, and the confidence interval excludes zero. The result looks ready to ship.

Then the team expands the analysis. Search activity falls by almost 15 points, browse is unchanged, and total activity per user remains flat. The treatment did not create more engagement. It changed where that engagement happened.

The home surface gained activity, but the product as a whole did not.

Nothing about the local calculation is wrong. The A/B test correctly answered, “Did activity increase on home?” The mistake was treating that answer as proof that total activity increased. A precise local estimate can still support the wrong launch decision when users, suppliers, or product surfaces compete for a shared resource.

Teams often respond by reaching for a switchback, cluster randomization, or separate budgets. Before changing the design, answer two simpler questions:

  1. Did the treatment increase the system total or redistribute an existing outcome?
  2. Were treatment and control isolated at the level where the shared resource operated?

This article presents a practical workflow for answering both questions. It reconciles local changes against a system total using the randomized-user denominator, then screens for coupling inside the resource pools shared by the experiment arms.

Why Shared Constraints Change the Meaning of Lift

An ordinary user-level experiment assumes that one user’s assignment does not change another user’s outcome. This condition is part of the Stable Unit Treatment Value Assumption (SUTVA). It becomes questionable when experimental units compete through a shared constraint:

Stable Unit Treatment Value Assumption


The first three examples can create interference between experimental units. The fourth can create cannibalization between product surfaces even when user assignment itself is clean. These mechanisms are related, but they are not the same claim:

  • Interference means that treatment assigned to one unit changes another unit’s outcome.
  • Cannibalization means that a measured gain comes from a corresponding loss elsewhere in the system.
  • Incrementality means that the treatment increases the outcome the system ultimately values, rather than changing its location or attribution.

Assignment, sample-ratio, exposure, and telemetry checks still come first. Once they pass, the next review asks whether the success metric covers the system outcome and whether the randomization unit isolates the rollout effect.

Use a Local-to-System Reconciliation

The local-to-system reconciliation brings those checks into one review. It has five parts:

  1. Define the shared constraint and system boundary. Identify the supply, inventory, budget, attention, or capacity through which outcomes can move, and decide what must be counted for the launch decision.
  2. Measure each local change. Estimate treatment-control differences for the components inside that boundary.
  3. Reconcile the total. Aggregate those components with one denominator tied to the randomization unit.
  4. Check isolation inside shared pools. Examine whether the arms move together inside markets, regions, seller clusters, or budget groups.
  5. Classify the evidence. Treat the result as incremental, redistributive, isolation-sensitive, or inconclusive before deciding whether to ship or redesign.

Decision flow from local lift through system reconciliation and isolation review

Figure 1: Decision flow from local lift through system reconciliation and isolation review

Start with the boundary. It must correspond to the business decision, and every component inside it must use a compatible unit of analysis. A convenient total that excludes the surface likely to lose value simply moves the blind spot.

Reconcile Outcomes per Randomized User

Here is what the reconciliation looks like in a deterministic synthetic experiment with three mutually exclusive surfaces:
Reconcile outcomes per randomized user

The home result is real. It just does not show incremental engagement: search loses almost exactly what home gains.

Calculate the treatment-control change for every affected surface using the randomized-user denominator. The system change is the sum of those component changes. The total absolute movement shows how much behavior shifted across surfaces. When movement is large, but the system change is close to zero, the treatment is primarily redistributive.

The implementation preserves the randomized-unit denominator:

Python
 
arm_size = df.groupby("variant")["user_id"].nunique()

surface_per_user = (
    df.groupby(["variant", "surface"])["activity"]
      .sum()
      .unstack(fill_value=0)
      .div(arm_size, axis=0)
)

system_per_user = surface_per_user.sum(axis=1)
surface_delta = (
    surface_per_user.loc["treatment"]
    - surface_per_user.loc["control"]
)

system_delta = surface_delta.sum()
local_movement = surface_delta.abs().sum()


Counting event rows instead would let an engagement-changing treatment alter its own denominator. Per-user outcomes preserve the connection between the reported effect and the randomization design.

Figure 2: Control and treatment have different surface shares while total outcome remains flat.

What To Watch For

  • Do not interpret shares without levels. A surface share can fall while its absolute activity rises if the total grows.
  • Make the parts add up. Overlapping or incomplete surfaces do not produce a clean system total. State what is excluded, and avoid double counting.
  • Keep the statistical review. Reconciliation is a diagnostic, not a standalone hypothesis test. Component and system estimates still need confidence intervals that respect the experiment design.

Use the system total as part of the Overall Evaluation Criterion (OEC): the metric or combination of metrics used to judge experimental success and connect short-term measurement to longer-term goals (Kohavi, Tang, and Xu, Chapter 7). Home activity explains where behavior moved; it should not decide the launch when the objective is total engagement or marketplace value.

Once the metric boundary is clear, the next question is whether the experiment arms were truly isolated.

Check for Coupling Inside Shared Resource Pools

Reconciliation can show that value moved. It cannot tell you whether treatment and control affected one another. A useful risk screen asks whether both arms remain related inside pools that share supply, inventory, or budget.

Do not correlate raw market means. Large markets can have high outcomes in both arms for stable reasons unrelated to the experiment. Remove each market’s pre-period baseline and compare changes:

Python
 
market_means = (
    df.groupby(["market_id", "period", "variant"])["revenue"]
      .mean()
      .unstack(["period", "variant"])
      .dropna()
)

control_change = (
    market_means[("post", "control")]
    - market_means[("pre", "control")]
)
treatment_change = (
    market_means[("post", "treatment")]
    - market_means[("pre", "treatment")]
)

coupling = control_change.corr(treatment_change)


The synthetic data returns a correlation of 0.873. Treat that as a review signal, not an “interference detected” verdict. Common shocks, seasonality, measurement changes, and omitted market characteristics can produce the same pattern.

The shared-pool coupling screen has one job: challenge the randomization unit. Define resource pools before looking at results, report the number and size of those pools, and add uncertainty or randomization inference. Even with those safeguards, the screen can identify risk in the original estimate; it cannot correct that estimate.

Convert the Evidence Into a Launch Decision

At this point, the team has two different forms of evidence: whether value moved across the system and whether the experiment arms may have shared the same constraint. Together, they lead to different actions:

Converting the evidence into a launch decision


The categories should not be collapsed into a single reliability score. Redistribution and interference answer different questions. A treatment can reallocate activity across surfaces without contaminating control, or it can interfere through shared supply even when the measured system total is positive.

Only an isolation problem necessarily requires a new experiment design.

Match the Design to the Constraint

  • Shared supply over time and region: Use a switchback to randomize the market state rather than individual transactions. DoorDash uses this pattern for dispatch experiments (DoorDash Engineering).
  • Fixed campaign budgets: Separate the budgets used by treatment and control. LinkedIn describes this budget-split design for its advertising marketplace (LinkedIn Engineering).
  • Marketplace or network spillovers: Randomize clusters rather than individuals, while accounting for the loss of power that comes with fewer independent units (Holtz et al.).
  • Competition between listings or surfaces: Keep local metrics as diagnostics, but base the decision on incremental platform value. Airbnb describes this distinction when accounting for booking cannibalization (Airbnb Engineering).

These are responses to a diagnosed constraint, not a checklist to apply to every experiment.

Key Takeaways

  • Name the shared supply, inventory, budget, or attention constraint before choosing the randomization unit.
  • Define the system boundary before reviewing local lift.
  • Calculate every component with the randomized-unit denominator.
  • Compare absolute outcomes before interpreting changes in surface share.
  • Treat shared-pool coupling as a design-review signal, not proof of interference.
  • Redesign only when the evidence shows that treatment and control were not isolated at the level where the system operates.

A p-value can tell you whether the observed treatment-control difference is compatible with a null hypothesis under a statistical model. It cannot tell you whether the experiment measured the right boundary or isolated the right unit.

The final question is not whether the treatment won its local comparison. It is whether launching the treatment would make the system better.

Reference Implementation

The deterministic example behind the reported values is available in the runnable Python script. A broader open-source implementation of related diagnostics is available in the Expguard repository and its archived release. These implementations are optional; the workflow applies independently of either codebase.

References and Further Reading

  1. Kohavi, R., Tang, D., and Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020.
  2. DoorDash Engineering, Switchback Tests and Randomized Experimentation Under Network Effects.
  3. Holtz et al., Reducing Interference Bias in Online Marketplace Pricing Experiments.
  4. Li et al., Interference, Bias, and Variance in Two-Sided Marketplace Experimentation.
  5. LinkedIn Engineering, Budget-Split Testing: A Trustworthy and Powerful Approach to Marketplace A/B Testing.
  6. Liu, Mao, and Kang, Trustworthy Online Marketplace Experimentation With Budget-Split Design.
  7. Airbnb Engineering, How Airbnb Measures Future Value To Standardize Tradeoffs.
Inventory (library) methodologies Testing

Published at DZone with permission of Arun Thomas. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Demystifying Event Storming: Process Modeling Level Event Storming (Part 2)
  • Importance of VLSI Design Verification and Its Methodologies
  • 5 Vital Steps in Successfully Setting Up Your Startup QA Process
  • 7 Failure Modes Every Production AI Agent Should Be Tested Against

Partner Resources

×

Comments

Loading comments

The comments didn't load

Please try again in a moment.

Likes

Loading

The next likes didn't load. Please try again.

The likes didn't load

Please try again in a moment.

See who liked this

Log in or join DZone for free to see who liked this article.

Log in Join for free
  • RSS
  • X
  • Facebook

ABOUT US

  • About DZone
  • Support and feedback
  • Community research

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 215
  • Nashville, TN 37211
  • [email protected]

Let's be friends:

  • RSS
  • X
  • Facebook