How a Consensus Score of 0.82 Is Computed: Quantifying Multi-Model Peer Review

Open Council has several AI models answer one question, then peer-review and score each other, and finally outputs something like consensus: 0.82. That 0.82 isn’t a hand-wave; it’s a number you can take apart. This post is about how it’s computed, and why “low consensus means no forced conclusion.”

The punchline: 0.82 is two halves put together

The core formula is a single line (consensus.ts):

rawAgreement = 0.5 * (1 - sigmaAvg / 4.5) + 0.5 * W

Two equally weighted halves:

• First half, score dispersion. Each answer is scored 1–10 by multiple reviewers; take the standard deviation sigmaAvg of those scores. The closer everyone’s scores, the smaller the deviation, the higher this term. Dividing by 4.5 is because the theoretical max standard deviation on a 1–10 scale is about 4.5, mapping dispersion into 0–1.
• Second half, ranking agreement. Kendall’s W measures “how consistently all reviewers ranked this batch of answers.” If everyone ranks the same answer first, W approaches 1.

So a typical 0.82 translates directly to: “score dispersion contributed about half, ranking agreement contributed the other half.” One number, two independent pieces of agreement evidence, not propped up by a single metric.

A deliberate division of labor: the dispersion half uses raw 1–10 scores (a standard deviation with interpretable units), while the Kendall’s W half first z-score-normalizes per reviewer (removing the “some rate loosely, some strictly” habit difference), then compares rankings. Same data, two lenses, each to its strength.

Small samples should be honest, not force a number

Fewer than 2 valid reviews returns a flat 0 — giving a confident consensus on an insufficient sample is a lie. With fewer than 3 reviewers, it also multiplies a correction factor rho (with 2 reviewers, rho = 0.5) to press down small-sample overconfidence.

The bug that nearly made the debate never stop

This is the one most worth writing about.

The consensus was originally also multiplied by a diversity factor delta: the fewer provider families involved, the lower delta (a single vendor takes a further hard 30% cut). It measures “how independent these reviewers really are” — all models from one vendor reviewing each other, agreeing, isn’t very trustworthy.

Early on, this delta got multiplied straight into the score used to decide whether the debate should stop. And it broke: in the single-model, multi-role scenario (only one model configured, split into 3 roles reviewing each other), delta = (1/3) × 0.7 ≈ 0.23. Even if the three roles reviewed in perfect agreement (full agreement score), multiplying by delta leaves only 0.23, never reaching the stop threshold — the debate mathematically can never stop.

The root cause is sharp: delta is a credibility discount factor (are the reviewers independent), yet it was jammed into the stopping score (do the reviewers agree). These two things are semantically orthogonal and shouldn’t be multiplied together before deciding to stop.

The fix was to separate the two scores completely:

• agreement_score (= rawAgreement × rho, without delta): measures agreement only, used solely for stopping;
• consensus_score (with delta): measures credibility, used only for display.

Stopping uses the former, so a single-vendor panel can stop normally as long as the reviewers actually agree; display uses the latter, so in the single-vendor case consensus_score is naturally low, honestly telling the user “this conclusion lacks independence.” One score drives control flow, one score drives outward expression; forcing them into a single number makes both awkward.

Low consensus doesn’t stop early, and doesn’t refuse

The stop threshold is 0.6, up to 3 rounds. Each round computes agreement_score; reach 0.6 and it wraps, otherwise it enters the next round of cross-examination — feeding disagreement points back so the models challenge each other, then re-reviewing and recomputing.

Where do the disagreement points come from? Besides aggregated weaknesses and the risks flagged by the “devil’s advocate,” it also looks per dimension: if any dimension (accuracy / completeness / practicality / insight) has a score standard deviation over 1.5, it explicitly pushes a “experts disagree significantly on ” back into the next round.

Note the stance: on low consensus the system doesn’t stop early (forces another round), but still produces a conclusion in the end (doesn’t refuse) — it just surfaces the lower consensus_score so you judge the credibility yourself. It would rather spend another round than rubber-stamp a low-agreement answer.

Score parsing: distinguish “no data” from “neutral data”

The reviews are natural language; scores have to be extracted from them. Parsing has three fallback tiers: try JSON first, fall back to regex grabbing each dimension, and on a final failure hand back a “filler” — a neutral 5 across all dimensions.

The key discipline: the filler fake 5s are marked parse_error, and the consensus computation only includes valid and partial reviews. What regex rescued (with real extracted scores) counts; the pure-placeholder fake 5s don’t enter the stats. Why be picky about this? Because a pile of neutral 5s artificially lowers dispersion and fabricates a “everyone agrees.” A parse failure is “no data,” not “a neutral data point” — treat the two as the same and you poison the stats.

Takeaways

Compressing “several models’ opinions” into a trustworthy consensus:

• 0.82 is two halves: half score dispersion (raw-score standard deviation), half ranking agreement (Kendall’s W), equally weighted;
• Small samples stay honest: under 2 returns 0, fewer than 3 reviewers multiplies a rho correction, no forced confidence;
• Control score and display score split apart: stopping uses the diversity-free agreement_score, display uses the discounted consensus_score — merging them makes both awkward;
• Low consensus neither stops early nor refuses: below 0.6 it feeds disagreements back for another round, and finally surfaces the true credibility for you to judge;
• Don’t let fake data into the stats: mark and exclude the neutral 5s from parse failures — distinguish “no data” from “neutral data.”

In one line: this system’s stance throughout is to preserve signal and explicitly annotate uncertainty, rather than hide or fabricate a certain conclusion.