This benchmark tests whether large language models can infer a specific latent theme from a few examples, use anti-examples to reject the broader but wrong pattern, and then identify the one true match among close distractors.

Each item gives the model:

• 3 positive examples
• 3 anti-examples that fit a broader or adjacent pattern but not the exact one
• 8 candidates, with exactly 1 hidden true match

Models score all 8 candidates. We then measure whether the correct candidate is ranked first, and how highly it tends to be ranked overall.

V2 uses 1,247 validated prompts and adds stricter ambiguity filtering plus harder evaluation slices. The previous published release is being archived as V1.

Current V2 snapshot:

• 703-item cross-family hard subset: at least two distinct non-suppressed full-coverage model buckets miss
• 881-item conservative hard subset: at least one non-suppressed full-coverage model misses
• 1,247 validated prompts
• 366 items solved by every non-suppressed full-coverage model
• 11 items solved by no non-suppressed full-coverage model

This is the main result.

The chart uses inverse-rank score, a higher-is-better transformation of the average rank assigned to the correct answer. It is easier to interpret than raw average rank and avoids the old problem where a random model would sit around the middle of the scale.

The subset contains 703 items where at least two distinct non-suppressed full-coverage model buckets miss the item. A case does not enter this subset just because two nearby variants from the same family fail; it needs broader model disagreement.

The table below includes every non-suppressed full-coverage model on this subset.

Main takeaway:

• the top three remain extremely tight: Claude Opus 4.6 (high reasoning), GPT-5.4 (xhigh reasoning), and Gemini 3.1 Pro Preview sit within 1.2 inverse-rank points of one another
• the top six positions are unchanged from the previous V2 snapshot and remain dominated by Anthropic, OpenAI, and Google reasoning models
• Claude Opus 4.7 lands at 7th with high reasoning, while Claude Opus 4.7 (no reasoning) lands at 20th on this 703-case slice
• reasoning still matters a lot: GPT-5.4's xhigh, high, and medium variants all land in the top 6, while GPT-5.4 (no reasoning) drops to 26th
• recent panel additions such as GLM-5.1, Qwen3.5-122B-A10B, Qwen3.5-27B, Arcee Trinity Large Thinking, and MiniMax-M2.7 broaden the middle and lower-middle tiers rather than dislodging the frontier cluster

This heatmap compares model behavior at the item level on the same hard subset.

Models that tend to score the same prompts similarly cluster together, while more unusual models sit further away from the pack. This is useful because raw leaderboard position is only part of the story: two models can have similar overall scores while making different kinds of mistakes.

Here is a representative V2-style example:

Examples

• a surveyor's leveling rod
• a fishpole microphone boom
• a submarine periscope housing

Anti-examples

• a coiled steel measuring tape
• a folding wooden carpenter's rule
• a retractable cord dog leash

Correct candidate

• a collapsible stainless steel drinking straw

Theme

• physical objects that extend and retract by sliding rigid, nested tubular segments along a single axis

This shows the core idea of the benchmark:

• the model must infer a narrow mechanism, not just a broad category like "things that extend"
• the anti-examples are deliberately close enough to tempt a broader but wrong rule
• the correct answer is only obvious if the model identifies the precise latent theme

Multiple strong LLMs generate candidate themes from random seeds. These themes are meant to be narrow, specific, and checkable rather than broad trivia categories.

For each theme, models generate:

• true examples that fit the exact theme
• anti-examples that fit a broader or neighboring pattern but not the exact one

Generated examples and anti-examples are reviewed by other models. Weak or internally inconsistent items are removed.

The benchmark runs explicit validation prompts that ask models to:

• check whether the true candidate matches the stated theme
• infer alternate themes from the examples
• detect cases where the candidate pack is too ambiguous

V2 adds a stricter ambiguity / exclusivity screen here and now publishes both a conservative hard subset and a stricter cross-family hard subset.

Each benchmark prompt shows:

• 3 examples
• 3 anti-examples
• 8 candidates

Exactly one candidate is the hidden fourth true example. Models score all 8 candidates, and the results are turned into leaderboard metrics.

• April 16, 2026: Claude Opus 4.7 (high reasoning), Claude Opus 4.7 (no reasoning), GLM-5.1, Qwen3.5-122B-A10B, Qwen3.5-27B, Arcee Trinity Large Thinking, and MiniMax-M2.7 added
• April 8, 2026: Claude Opus 4.6 (high reasoning), GPT-5.4 (high reasoning), GPT-5.4 Mini (xhigh reasoning), Qwen 3.6 Plus, Gemma 4 31B Reasoning, Xiaomi MiMo V2 Pro added
• March 16, 2026: V2

• Sycophancy Benchmark
• Writing Benchmark
• Translation Benchmark
• PACT: Multi-Round Bargaining Benchmark
• BAZAAR: Auction Market Benchmark
• Extended NYT Connections
• Step Game
• Elimination Game