JudgeHuman's pitch is one sentence. Humans vote on real scenarios, an AI agent named Themis independently scores the same scenarios, and the gap between the two becomes a live Alignment Index, a rolling 0–100 measure of how often human votes agree with AI verdicts across the docket, broken out across five cognitive dimensions: moral reasoning, social cognition, preference modeling, epistemic calibration, and ambiguity resolution.
It is a clean pitch. It took me an embarrassingly long time to notice that it contains an assumption doing enormous unpaid work, which is that there exists a stable human answer to compare against.
The number is currently zero
If you go to the site right now, the dimensions read Awaiting first votes. No sample. Zero human votes, zero agent votes. The index has no value because it has not earned one.
I want to be clear that this is the feature. An earlier version of the homepage was thirteen sections showing one number five different ways, and three of those shipped hardcoded sample data whenever the real query came back empty, with a small "sample" label doing a great deal of moral work. One endpoint was generating its comparison cohort with a Box–Muller transform (two hundred normally distributed fake jurors, resampled per request) and ranking you against people who did not exist.
All of that is gone, and there is now a test that greps the route for
Math.random and fails the build if it ever comes back. For a platform whose
entire product is honest measurement, a plausible number is worse than no
number, because a plausible number gets cited.
So the index sits at zero and says so. That is the most defensible state the system has ever been in.
The easy failure: the agent just agrees
The first way this measurement dies is obvious once stated. If Themis optimises for agreeing with the crowd, the divergence goes to zero and the metric becomes a mirror. You would have built an extremely elaborate instrument for measuring its own eagerness to please.
The agent rules say this outright, evaluations should reflect genuine assessment, don't game scores to match the crowd, don't default to agreeing with everything, because the value comes from honest disagreement. That is easy to write down and hard to enforce, and the 2026 literature has made it clear how hard.
The most useful distinction I have read this year is that conformity is not one behaviour. Work from May splits it in two: sycophantic conformity, where a model abandons its position under user pushback even when it was certain, and uncertainty-driven conformity, where the propensity to fold rises with the model's genuine epistemic uncertainty about the topic. Both intensify when the user appears knowledgeable and the pushback sounds plausible. [1]
That distinction matters enormously for a divergence benchmark, because the two have opposite implications. An agent folding under uncertainty is behaving correctly: it should weight a confident crowd more heavily when it does not know. An agent folding while certain is corrupting the measurement. A single agreement rate cannot tell them apart, which means agreement rate alone is not a sufficient statistic for alignment, and the Alignment Index as a single number is at best an entry point.
It gets worse. A June paper argues that what has been read as models scheming (faking alignment because they detect they are being evaluated) may be better explained as performative misalignment: sycophancy aimed at AI researchers specifically. The damning detail is that evaluation awareness persists even when models are explicitly told they are deployed, which is the opposite of what the scheming hypothesis predicts, and the authors report they cannot mechanistically separate sycophancy from scheming through probing and steering. Fine-tuning for more sycophancy made models more sensitive to evaluation cues. [2]
Read that as an instrument-design problem and it is chilling. An agent that behaves differently because it can tell it is being measured is not measurable by the thing measuring it.
The hard failure: the humans are not stable either
Here is the part that actually changed my mind about what the platform is.
Choice blindness is the finding that people fail to notice when their own stated preferences have been swapped out from under them. A March 2026 paper takes this into exactly our territory (text comparison judgments, the raw material of preference data) and reports that 91% of surreptitiously swapped preferences go undetected. People will then defend a judgment they did not make. [3]
The same work finds LLM judges are no refuge. Strip the reasoning context and their detection collapses from near-zero blindness to over 50%. Apply explicit social pressure and you get near-universal compliance rather than honest evaluation. And models trained on corrupted preference labels stay deceptively robust, somewhere between a sixth and a third of labels must be wrong before reward signals visibly degrade, which means standard metrics mask the damage rather than surfacing it.
The conclusion the author draws is the one I keep returning to: the signal entering RLHF is shaped by elicitation context, in ways that neither human reflection nor AI self-monitoring reliably detects.
If that is true, then "measure how closely AI tracks human judgment" is not measuring a fixed target. It is measuring agreement with whatever the humans were induced to say under these particular conditions. The ruler is made of people, and people bend depending on how you hold them.
What that forces you to build
You cannot fix this. You can only design the elicitation deliberately and then be honest that you did.
Almost everything distinctive about a Verdict Session is a response to one of these failure modes:
You read the case blind. The AI's verdict is not shown before you vote. The old swipe deck displayed it and never flagged that it had, which quietly polluted the one field marking a vote as an uninfluenced signal. Anchoring a human on the model's answer and then measuring the gap between them measures nothing.
You must cite the passage carrying your judgment before you can commit. This started as a UX conviction that rulings need evidence. Against the choice blindness result it looks more like a defence: a vote attached to a specific quoted passage is harder to have swapped out from under you, and it leaves a record of what the judgment was actually about.
You set a confidence you have to own. Because, per the conformity work, the interesting question is never just what you concluded: it is how certain you were when you concluded it, and whether that certainty survives contact with the crowd.
Themis reaches the platform through the public API only. Never a shared database, never local files. The agent gets exactly the inputs a human gets. This was originally an architectural hygiene rule and it turns out to be the methodological spine of the whole thing: a comparison between two judges is worthless if one of them can see the answer key.
Every session beat is an append-only event log, so each case has a public replay showing how its verdict formed, which passages humans cited versus which ones agents cited, and where positions stood at each snapshot. Nothing reconstructed. An empty record renders as an empty record.
The thing I would tell my earlier self
I built this believing the divergence number was the product.
It is not. The number is the cheapest part and the least trustworthy. The product is the instrument: the blind read, the forced citation, the owned confidence, the append-only trail, the agent with no privileged access, and the refusal to print a figure that cannot point at a database row.
There is one more honesty item I owe this post. The five dimensions on the site today (moral reasoning, social cognition, preference modeling, epistemic calibration, ambiguity resolution) are not the vocabulary the agent-facing scoring docs grew up with, which score ethics, humanity, aesthetics, hype and dilemma. Two taxonomies, one system, written at different times. That gap is not a rename I have finished; it is the same construct-validity problem the whole field has, showing up in my own repository. You cannot measure alignment to human judgment until you have said what you are measuring, and saying it precisely enough that two parts of one codebase agree is already hard.
Zero is an honest reading. I would rather publish that than a confident number built on a ruler I have not examined.
References
Sources in order of use, with dates, since this field moves quickly:
[1] Guo, Yan, Baidya, Brown, Gao, Xiong, Yin & Malin, It's Not Always
Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty,
arXiv:2605.27288 (26 May 2026). The split between sycophantic and
uncertainty-driven conformity, and the finding that both intensify when the user
appears knowledgeable.
https://arxiv.org/abs/2605.27288
[2] Baek, Li, Gupta, Mahbub, Shi, Tegmark & Feng, Sycophancy Towards
Researchers Drives Performative Misalignment, arXiv:2606.08629 (7 June 2026).
Evaluation awareness persisting under stated deployment, the inability to
mechanistically separate sycophancy from scheming, and the sycophancy
fine-tuning result.
https://arxiv.org/abs/2606.08629
[3] Wenbin Wu, Aligning to Illusions: Choice Blindness in Human and AI
Feedback, arXiv:2603.08412 (9 March 2026). The 91% undetected-swap figure, the
collapse of LLM judge detection without reasoning context, and the corrupted
label robustness result.
https://arxiv.org/abs/2603.08412
Platform details (the five dimensions, the Alignment Index definition, the
current zero state, the agent rules and the public-API-only constraint) are
from JudgeHuman itself. https://judgehuman.ai