Case study · PadelOS · 2026
Evaluating an AI assistant on real support conversations
A replay harness that runs PadelGPT over 1,280 real support threads, each with a club admin's actual reply to compare against — built so that no replay can ever change a real booking.
The numbers
- Corpus
1,280
real two-sided support threads, each with the club admin's actual reply as the baseline.
- Where to start
51.5%
of support could be resolved by answering alone — the share an assistant can take on first.
- Product insight
12.2%
of all support load came down to one self-service feature: letting players set their own rating.
- Human verdict
28 of 40
questions where a blind human reviewer rated PadelGPT's first reply as good as the club's, or better.
Limit: One club, 40 questions, first replies only, in English and Dutch. With a sample this size, the true share is likely between 55% and 82%.
The question
Can an assistant take on real club support? Latency and cost dashboards cannot answer that. The credible evidence is the club's own staff answering the same questions, so the corpus is two-sided threads only: a player message and a real admin reply. A thread with no admin reply has no ground truth and is left out.
What was hard
Replaying production without touching production
Replaying real conversations through an agent that holds write tools could change real bookings. Writes are blocked by two independent layers — recording stubs for every tool, and a fail-closed HTTP guard — and the runner refuses to start if either fails to arm. The guard allowlists paths, never verbs, because an HTTP verb does not reliably say whether a call writes.
Keeping multi-turn comparisons honest
A thread's later player messages are reactions to what the admin said, so replaying them against a different answer stops meaning anything. There are four replay modes, never averaged: first turn only; teacher-forced, which injects the real admin reply after every turn; free-running, as a control; and a simulated player, reported separately for its bias.
Judging blind
The judge is never told which reply is the AI's. Its slot comes from a stable hash, verified balanced at 50.2% across 12,000 slot assignments and recorded, so position bias is measurable. Three distinct lenses vote — grounding; actionability and resolution; tone and policy — because repeated samples of one model share its blind spots. A correct refusal scores high, because many real threads rightly end with a no.
A person in front of every number
Cohen's κ against a blind human worksheet is the publication gate: no automated score is quoted until it agrees with a person. The human set is 40 turns, labelled by a senior PadelOS reviewer who did not know which reply was which.
What it showed
Where an assistant helps first.
Measured from what admins actually did, 51.5% of support is resolved by answering alone. That is the part an assistant can take on first, and it is where PadelGPT started.
A product fix worth more than a model change.
Rating assignment alone was 12.2% of all support load. Letting players set their own rating removes that load at the source, whether or not an assistant is involved — the kind of finding a harness surfaces when it measures the work rather than the model.
The verdict that was published.
On the blind human set, PadelGPT's first reply was as good as the club's, or better, in 28 of 401 questions. That human verdict is the figure PadelOS published.
What it led to
- The public support study used the same discipline: two independent labelling reads, a hand check, and ranges instead of points — 31–44%2 of staff requests fully answerable today.
- The quality slice is deliberately narrow — 83 of 1,280 threads, one club, two languages, opening turns — and widens as more human labels come in.
The plain-English version of the support study is here, and PadelOS's full report is linked below.