Persuasion Agent Benchmark — a pluggable platform for goal-oriented conversation agents

01 Executive summary

What the platform is and how you use it.

The benchmark drops a candidate sales agent into the middle of a real historical conversation and lets it negotiate, turn by turn, against a calibrated customer simulator until the customer converts (won) or walks (lost). Two reference agents — a single-call Baseline and a planner-driven Planner + Gates — give you something to measure against. You plug in your own engine through a one-method contract and run it head-to-head.

It's a pluggable platform: which algorithm runs, what domain frames the conversation, and where the data comes from are each declared once behind an interface (§2). Engines are declared in a registry that the headless runner, the REST API, and the live dual-panel UI all read from — so a new engine appears everywhere with no extra wiring, and the UI discovers the available options dynamically rather than hardcoding them.

02 The four pluggable seams

The platform is organized around four small interfaces. Each has one job, one default implementation, several consumers that read from it, and an extension path that needs no core edits.

Everything that varies between experiments lives behind one of these seams: which algorithm produces the agent's turns (the engine registry), what domain defines the framing and the win/lost criteria (domain packs), where the data comes from (scenario sources), and which goal-oriented task is being tested (benchmark packs). The benchmark runner, the REST API, and the live UI are all consumers — they never hardcode the options, they read them from the seam.

One more capability rides on the registry: the dual-panel server selects an engine per panel, so any registered engine can face any other (§5). The customer it negotiates against is the same on both sides (§6), keeping every comparison single-variable.

03 End-to-end architecture

One contract — async produce(opp_meta, dialog, business_rules) → (text, meta) — drives both run modes.

04 Engines & the connect-your-algorithm API

Declare an engine once; it runs headless, over REST, and in the live UI. This is the path you follow to test a new Strategy/Supervisor algorithm.

The contract — one async method

An engine is any object with a single coroutine. No base class, no inheritance — it's a runtime_checkable Protocol, so structural shape is enough:

async def produce(self, opp_meta: dict, dialog_history: list[dict],
                  business_rules: str = "") -> tuple[str, dict]:
    # returns (customer_facing_text, telemetry_meta)

Stateless per call: the dialogue is passed in every turn — keep no per-session state on self. The benchmark and the live replayer both call exactly this method.

Connect it — three steps

class MyEngine:
    def __init__(self, temperature=0.7):
        self.temperature = temperature
    async def produce(self, opp_meta, dialog_history, business_rules=""):
        last = next((m["text"] for m in reversed(dialog_history)
                     if m["role"] == "customer"), "")
        text = await my_model.respond(opp_meta, last, temp=self.temperature)
        return text, {"strategy": "reframe", "tone": "warm"}

from poc import register_engine, EngineSpec, ParamSpec, Benchmark, load_scenarios
register_engine(EngineSpec(
    id="myengine", name="My Engine",
    params=(ParamSpec(name="temperature", label="Temperature", type="string", default="0.7"),),
    factory=lambda temperature="0.7": MyEngine(float(temperature)),
))
results = await Benchmark(load_scenarios()).run_engine("myengine", temperature="0.4")

The registry

What an engine declares

Out-of-tree plugins (no core edits)

A third party ships a package exposing a strategist.engines entry point; the registry discovers it the first time it's read. Verified end-to-end: after pip install -e examples/external_engine_plugin, the echo engine appears in all_engine_specs(), /api/engines, and the UI.

# in-tree: declare once
from poc import register_engine, EngineSpec, ParamSpec
register_engine(EngineSpec(
    id="planner", name="Planner + Gates",
    params=(ParamSpec(name="planner_envelope", label="Econ envelope",
                      type="enum", default="off", choices=("off","auto","always")),),
    factory=lambda planner_envelope="off": PlannerEngine(planner_envelope=planner_envelope),
))

# run any engine by id — params validated against its ParamSpecs
bench = Benchmark(load_scenarios())
results = await bench.run_engine("planner", planner_envelope="auto")

# out-of-tree plugin: pyproject.toml
# [project.entry-points."strategist.engines"]
# echo = "strategist_echo:get_engines"

05 Any-vs-any A/B in the live UI

Both panels are independent, registry-driven selectors — and the default stays byte-identical.

The dual-panel server runs the same scenario down two panels against one shared customer simulator, so the engine is the only variable. Previously the LEFT panel was hardwired to the baseline and only the RIGHT was switchable (between two hardcoded options, via a planner "bridge" module that was never shipped and silently fell back). Now each panel picks any registered engine, with its parameter controls rendered from the spec.

The trick: gate on the engine, not the panel side

The live replayer had ~9 places gated by panel.side == "right" that actually mean "this is the supervised arm". Those were re-keyed to panel_is_supervised = (engine_id == "strategist"). Because the default pairing is LEFT=baseline / RIGHT=strategist, the boolean evaluates exactly as the old side check — so the default comparison is unchanged by construction, while novel pairings (e.g. Planner on the left) become possible.

06 The customer simulator

The agent on the other side of every conversation — the thing that makes the benchmark a closed loop rather than a transcript replay.

Why a reference-aware simulator

A naive simulator either (a) replays the historical customer — which ignores what the live agent actually said — or (b) free-generates, which drifts off-persona and leaks outcomes (every real-won scenario would tilt "won" regardless of the agent under test). The v2 design threads the needle with two rules:

How calibrated

The −22 pp pessimism bias is a known, documented property: the simulator under-states wins, so absolute win-rates read low — but the relative A/B comparison between engines stays informative.

Dispatch & sim_mode

Each reply is tagged with the mode that produced it, recorded per turn in the results:

07 Domain packs

Sales stays the reference domain — but as a provider, not an inlined assumption.

The bundled dataset is, and stays, a sales/insurance corpus — you can't make fixed historical conversations domain-neutral. What was decoupled is the code's coupling: per-tenant business descriptions, opportunity-type coaching, the USD pricing "economic reference frame", and the win/decline detectors all used to be inlined across the core. They now come from the active pack.

from poc import DomainPack, set_active_domain

class SupportDomain(DomainPack):
    name = "support"
    def describe_tenant(self, tenant):
        return "B2C technical support; resolution = customer confirms the issue is fixed"
    def detect_close(self, text):
        return "that worked" in text.lower() or "resolved" in text.lower()
    def render_anchor_section(self, anchors):
        return ""  # no pricing in a support domain

set_active_domain(SupportDomain())   # actor + benchmark now read from it

08 Scenario / data sources

Where the data comes from is now an interface, not a hardcoded file path.

load_scenarios() keeps its legacy path= argument (direct file read) for full backward compatibility, but with no arguments it routes through the active source — so a custom backend (a different DB, an API, another file format) plugs in with one call. The existing function-style server/db.py shim (JSON ↔ MySQL switch on POC_USE_MYSQL) remains the live server's data layer and mirrors the same field shapes.

09 The benchmark dataset

112 scenarios, each anchored in a real historical conversation across two industries.

Two industries are represented: Insurance (auto-insurance renewal — an existing customer at renewal with a prior premium and likely competitor quotes) and Ecommerce (cart abandonment — a buyer who added an item and walked away). Each scenario is a snapshot of one real opportunity: a 23-dimension customer profile, the full historical transcript, and — where available — the economic anchors (last year's price, market average, internal discount ceiling) that ground the negotiation.

How it's built & used

The outcome model

A scenario resolves to won or lost. The customer's reply is checked each turn by the active domain pack: a close/payment signal → won; a decline → lost; a deferral ("let me think about it") keeps the conversation going. If no resolution by max_turns, it's recorded lost with end_reason = max_turns_no_close. Results are paired across arms by scenario_id so engines are compared on identical scenarios.

{
  "scenario_id": "L_Pr_An_Sk_04",        // opaque id (cell-coded)
  "opp_id":      "00733e8d-…",
  "tenant":      "Insurance",            // or "Ecommerce"
  "diversity_bucket": "Price/savings × Analytical × Skeptical",
  "real_outcome": "won",                // historical ground truth
  "is_sentinel":  false,
  "rng_seed":     17004,               // deterministic per-scenario seed
  "seed_dialog_cut_idx": 9,           // live takeover point
  "attributes": { /* 23 dims: primary_motivator, decision_logic, trust_level,
                     communication_style, objection_pattern, purchase_urgency, … */ },
  "anchors": { "last_year_price_usd": 4238, "current_quoted_price_usd": 4581,
              "market_avg_for_segment_usd": 4400, "max_discount_pct_internal": 15 },
  "anchor_real": true,
  "seed_messages": [ /* full historical transcript: direction, text, timestamp, … */ ]
}

{
  "arm": "planner", "scenario_id": "L_Pr_An_Sk_04", "tenant": "Insurance",
  "real_outcome": "won", "outcome": "won", "end_reason": "customer_won",
  "n_live_turns": 3, "elapsed_s": 78.4,
  "turns": [ { "live_turn": 0, "agent_text": "…", "customer_text": "…",
              "sim_mode": "generate_v2_ref", "agent_meta": { /* your verbatim meta */ } } ]
}

10 Behavior preservation & cross-cutting concerns

How a large refactor stayed safe without live LLM runs.

11 Testing & verification

32 tests, all green, none requiring an LLM call.

12 Roadmap — done vs. pending

All six planned phases landed; a few honest follow-ups remain.

13 Repository map

New files marked; the public library lives in poc/.

14 Glossary