Play the Match, Not the Exam

The exam has no answer key

Most AI benchmarks are multiple-choice exams. There's a fixed question, a hidden answer key, and a score that says how often the model picked the right box. That works beautifully for translation, classification, or "what's the capital of France."

A sales conversation is not an exam. It's a match. The moment your agent says something, the customer reacts — and the next question depends entirely on what just happened. There is no single correct reply to "your price went up and I found cheaper elsewhere." There are dozens of reasonable moves, and the one that works depends on who you're talking to and how the rally has gone so far.

So the classic recipe breaks down. You can't collect "ground truth" for a conversation that never happened, because the instant your agent diverges from the historical transcript, the rest of that transcript becomes fiction. The customer in the record was responding to a different agent.

Grading a persuasion agent against a fixed transcript is like grading a striker by replaying last season's match and checking whether they stand exactly where the old striker stood.

So we play matches

If there's no answer key, judge the play. We evaluate agents the way you'd evaluate a football team: put two of them on the same pitch, against the same opponent, under identical conditions — and see who wins, who wins head-to-head, and who plays the better game.

Concretely, that's an A/B test over synthetic dialogues. We take a real customer situation, hand it to two engines, and let each one negotiate a full conversation against a simulated customer. Same customer, same starting point, same everything — the only variable is the agent. Whoever closes the deal scores; and because we watch every turn, we can also see how they scored.

The opponent: a calibrated customer

A match is only as meaningful as the opponent. A sparring partner who folds at the first offer tells you nothing; one who behaves like a real, varied buyer tells you a lot.

So the other side of every conversation is a customer simulator — an LLM playing the buyer. The hard part is making it both realistic and fair. Two design rules do the heavy lifting:

We calibrated it against held-out real conversations. It agrees with the real outcome 78% of the time, and judges rate its adaptiveness 4.71 / 5. It also has a known, deliberate bias:

The playbook: Cialdini's six

If the match is the format, persuasion is the game being played — and the canonical playbook is Robert Cialdini's Influence. His six principles aren't decoration here: they're recorded as data on every single turn.

Each turn in our precedent corpus carries a flag for which principle the agent just used (reciprocity_activated, commitment_activated, and so on), and each customer carries traits for how susceptible they are to a given lever (authority, social_proof_susceptibility, …). Persuasion is literally the schema.

Cialdini later added a seventh principle, Unity (shared identity). It isn't encoded in our data yet — a natural next play to add to the book.

A worked rally

Here's a short rally on an insurance-renewal scenario — the customer opens skeptical and price-anchored. Watch the commitment level climb as the agent runs the playbook.

Notice there was no single "correct" line — a different customer (more socially-driven, less analytical) might have needed Social proof instead of Authority. That's exactly why a fixed answer key fails and a match succeeds: the right play depends on the opponent in front of you.

The pitch: our dataset

Every match is played on a real pitch. Our 112 scenarios are each anchored in an actual historical opportunity — not invented prompts — across two industries.

Each scenario is a snapshot of one buyer: a 23-dimension profile (their motivator, decision style, trust level, objection pattern, susceptibility to each Cialdini lever), the full historical transcript, and the economic anchors that ground the negotiation (last year's price, the segment's market average, the internal discount ceiling).

Why it's built this way

• Maximum variety, minimum size. The 15 cells are chosen by farthest-point sampling on the profile vector, so a compact set still spans wildly different buyers — analytical-skeptical, impulsive-trusting, and everything between.
• Self-contained matches. The historical transcript ships with each scenario, so the simulator's same-turn reference needs no database at run time. We seed the dialogue up to a takeover point, then the agent plays on.
• Sentinels. About a fifth are deals that genuinely closed in real life. They're a floor: an engine that fumbles the easy wins is regressing, no matter how it does on the hard cases.

Keeping score: the metrics

A final score alone is a noisy signal — one deflected shot can flip a 1–0. So we score the match at three levels, from the result down to the equivalent of expected goals.

That third level matters because conversion is rare and noisy. Win-proximity is our expected-goals: it reads the quality of play, so two engines that both "lost" a hard scenario are still distinguishable — one created chances, the other didn't. It's how we cut variance and see real differences without needing thousands of matches.

The rules of the match

An A/B test is only trustworthy if the two teams truly played the same game. Four rules keep it clean.

What a result looks like

Run two engines over the set and you get a paired summary: each engine's win rate, and the head-to-head tally.

{
  "arms": {
    "baseline": { "n": 112, "won": 56, "win_rate": 0.500 },
    "your_engine": { "n": 112, "won": 71, "win_rate": 0.634 }
  },
  "pairwise": { "your_engine_better": 19, "baseline_better": 4, "ties": 89 }
}

Illustrative shape, not a measured claim. The 89 ties are mostly scenarios both engines win or both lose; the story is in the 19-vs-4 split where they diverge.

What it means to win

In football, you can win ugly — but the teams you remember win well. The same is true here. The deal is the score, but the experience is the game.

An agent that closes by dropping its price five times in a row "wins" the deal and loses the customer's trust. So we watch for exactly that: a no-staircase rule flags repeated capitulating discounts, and we reward agents that hold a fair price, address objections with value, and — when a deal genuinely isn't there — close gracefully rather than spamming. Winning means converting and leaving the human better off than a competitor's agent would.

Your team's success metric isn't just the goal — it's winning while giving the human a better experience than the other side's striker would.

Step onto the pitch

The whole point of a match format is that anyone can field a team. Bring an engine, and it plays the same fixtures against the same opponent.

An engine is one method — given the customer profile and the conversation so far, return the next thing to say. Register it once and it runs in the headless league and the live, side-by-side viewer alike.

async def produce(self, opp_meta, dialog_history, business_rules=""):
    # read the customer + the rally so far, run your strategy…
    return reply_text, {"principle": "reciprocity", "tone": "warm"}

results = await Benchmark(load_scenarios()).run_engine("your_engine")

Then read the scoreboard: who scored more, who won head-to-head, and — by replaying the film turn by turn — who played the better game. No answer key required.