Grounded in real behavior
User profiles, visual observations, product context, and actions aligned at every step.
THE BENCHMARK FOR INTERACTIVE USER SIMULATION
UserArena.
Real users. Long journeys.
More faithful agents.
Benchmarking interactive user simulation for long-horizon shopping agents —
from the next click to behavior across sessions.
ONE USER, MANY VISITSA journey beyond
a single click.
a single click.
091011
Connected across sessionsSESSION 0901
Discover →
SESSION 0902
Consider →
SESSION 1103
Return →
BEHAVIOR, IN CONTEXT
Observe → Act → Evolve
clickadd to cartreturn
13 action types. One interaction loop.64,729Session trajectories
11,010Real users
566,102Products
577,670Recorded actions
01 / THE BENCHMARKBeyond predicting
Beyond predicting
the next click.
A faithful user simulator should preserve how people browse, decide, and return. UserArena connects real shopping behavior to an executable environment and real-user outcomes.
Built for the long horizon
Chronologically linked visits reveal whether a simulator preserves preferences over time.
Validated beyond the trace
Five evaluation levels connect behavior fidelity to controlled offline A/B outcomes.
Read the research summary
Agentic recommendation depends on recommendation world models that capture both the shopping environment and how users respond. UserArena pairs 64,729 real-world session trajectories with an interactive environment and controlled offline A/B outcomes. Its five-level evaluation protocol measures environment compliance, local decisions, sequential behavior, personalization, and outcomes. The paper also introduces On-Policy Trajectory Distillation (OPTD), which provides intent-guided teacher targets on student-generated rollouts, constrained by a per-token trust region and recorded behavior.
02 / INSIDE THE ARENA
Follow the journey.
Explore one shopper’s behavior across three consecutive visits. Inspect the recorded action sequence and selected visual observations.
VISUAL OBSERVATION
04
No presentation frame
This action is in the recorded sequence. An English reconstruction of this observation has not been added to the gallery.
THE RECORDED SEQUENCE9 actions
Brow tint & eye cream
Two products, two add-to-cart actions.
2 available frames
ABOUT THESE VISUALS
English presentation reconstructions with retouched product images and rescaled display prices. Four click observations are available; other steps retain their action labels. “Buy” is a logged action, not confirmation of payment.
THE NEXT STEP
Your agent. The shopping environment.
Interactive environment access and evaluation runs will be linked here at release.
03 / EVALUATION PROTOCOLFive levels.
Five levels.
A fuller picture of fidelity.
Choose a level to see what it measures. Replay evaluates the next decision under a real state; rollout evaluates behavior over an entire session.
ROLLOUTL2
Does the journey
behave like a real one?
Measure realistic decision patterns after product visits and the number of actions in each session.
04 / BENCHMARK RESULTS
The SOTA leaderboard.
Evaluate state-of-the-art LLMs as shopping simulators under one controlled protocol, from executable actions to long-horizon outcomes.
| Rank | Model | Family | RVA ↑L0 | OVA ↑L0 | F1 ↑L1 | Acc. ↑L1 | DM-JSD ↓L2 | TE ↓L2 | PF ↑L3 | ΔCTR ↓L4 | ΔACR ↓L4 | T-JSD ↓L4 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| No verified SOTA scores yet.Only complete Replay and Rollout reports are ranked. No scores are estimated. | ||||||||||||
05 / THE HUMAN PERSPECTIVECan you tell
Can you tell
who is human?
Beyond automatic metrics, the paper uses a source-blinded Turing-style evaluation. Annotators inspect two trajectories in a shared user context and identify the machine-generated one — or choose “Unsure.”
Anonymized interface example. English presentation copy; no platform or institutional branding.
THE METHOD / OPTDLearning from intent.
Learning from intent.
Staying grounded in behavior.
On-Policy Trajectory Distillation provides dense teacher supervision on student-generated rollouts. Trajectory-level intent guides learning; a per-token trust region and recorded behavior keep it grounded.
Student rollouts→Intent-guided teacher→Trust-region targets→Behavior grounding
06 / BUILD WITH USERARENA
Start exploring.
Data, the hosted environment, and community submissions will be linked when available.