Context is Social,
Tokens Aren't

We live in perilous times. But even so, we should always strive for greatness; that is what makes us human. We make mistakes, but we are capable of rising above them. And right now, one of the most consequential bets humanity is making is on artificial intelligence.

I am a student. I am not a senior researcher at a frontier lab. I don't have a team, a budget, or institutional backing. What I have is a conviction that AI is one of those technologies that will shape mankind, not in the abstract future-tense way people say at conferences, but right now, in the way people interact, make decisions, and understand each other. And I believe that the people who care about where this goes should be in the room where it's being built, not watching from the outside.

This benchmark was my attempt to put my ring in the hat.

As AI systems become more embedded in daily life (as chatbots, assistants, and agents), the quality of those interactions depends on something we rarely measure well: whether the model actually understands what you mean, in context, as a social being. Not just whether it produces grammatically correct and topically relevant text. Whether it understands you.

I kept noticing that there was an increasing reliance on AI as it becomes more capable. People trust these systems more. They defer to them more. And that trust is being built on something we haven't fully interrogated: the assumption that social fluency equals social understanding.

A model can sound like it understands you without actually understanding you. It can produce a confident interpretation of an ambiguous social situation when the honest answer is: I don't have enough to go on. The gap between those two things (generating socially plausible text versus actually reasoning about social meaning) is what I wanted to measure.

I designed a benchmark called Context is Social, Tokens Aren't: a two-task evaluation of social interpretation under ambiguity. The core idea is simple but the implications are not: socially competent interpretation requires two abilities, and most benchmarks only test one of them.

These two tasks belong together because they're two sides of the same coin. A model that interprets confidently when it should is useful. A model that interprets confidently when it shouldn't is dangerous confidently when it shouldn't is dangerous, especially as people rely on it more.

The dataset was built around matched context-sufficiency contrasts: pairs of items where the utterance family is approximately the same, but whether the context supports inference changes. This lets the benchmark isolate a specific failure mode: not just “did the model interpret correctly” but “did the model know whether it was in a situation where interpretation was warranted.”

I evaluated 24 models. The headline result is clean: models are systematically better at interpreting correctly than at knowing when not to interpret. The mean performance gap between the two tasks across all models was 11.1 percentage points, confirmed by a paired t-test (t = 8.482, p < .001). This isn't noise. This is a structural bias in how these systems handle social ambiguity.

But the more interesting story is in the directional failures.

These aren't just different points on the same spectrum. They're qualitatively different failure modes with different consequences. An overcommitting model tells you something wrong with confidence. An overcautious model refuses to help you when it could. Both break trust, but in different ways.

At the scenario level, the failures were just as organized. Underspecified commitment and minimal response ambiguity overwhelmingly produced overcommitment. Recoverable indirect requests and sufficient hedged refusals overwhelmingly produced over-caution. The benchmark wasn't just measuring model-level tendencies; it was revealing scenario-specific failure patterns that cut across model families.

The hardest items in Intent Inference were cross-task boundary cases and soft refusals: situations where socially similar acts had to be distinguished from one another. The hardest items in Clarify-or-Assume were the matched contrasts themselves: cases where the model had to decide whether the evidence actually licensed the kind of inference it was tempted to make.

I want to be careful here. I am a student who ran a benchmark on a Kaggle hackathon. I am not claiming to have solved anything. What I am claiming is that this failure mode is real, it is measurable, and it matters.

As AI systems take on more socially embedded roles (as assistants that help people navigate relationships, decisions, and interactions), the cost of overcommitment goes up. A model that confidently interprets an ambiguous situation as a rejection, when it was actually just a deflection, can damage a relationship. A model that refuses to interpret a clear indirect request wastes your time and erodes trust in a different way.

The practical implication is that systems deployed in ambiguous social settings should have clarification triggers: explicit mechanisms to recognize when the evidence doesn't support confident interpretation and to ask rather than assume. Uncertainty should be surfaced, not hidden. Confidence should be calibrated, not performed.

I started this project because I believe AI will shape mankind, and I believe that the people who care about where it goes should be building it, not just using it. I am not naive about what a student-run benchmark can accomplish. But I also believe that you start where you are, with what you have, and you put your ring in the hat.

The title of this benchmark is Context is Social, Tokens Aren't. What I mean by that is simple: the meaning of language is not in the words. It's in the situation, the relationship, the history, the power dynamics, the stakes. Models process tokens. Humans process context. The gap between those two things is where social intelligence lives, and where, right now, our most capable AI systems still fall short.

We are capable of building systems that do better. That's what makes this work worth doing.