Vals finds Fable 5.1 fully answers 26.3% of 519 immigration questions

• Fable 5.1 satisfied every expert criterion on just 26.3% of immigration questions, according to ValsAI.
• Published results cover 262 private test questions drawn from 519 expert-written scenarios, as ValsAI explains.
• Recent policy was the biggest blind spot: models met 7.3% of criteria concerning 2025 and 2026 rules, versus about 64% for established law in ValsAI's evaluation.

GPT-6 Luna averaged more than 100 web searches per conversation for a 1.8-point gain. Even the winning model lost to another model on 134 of the 262 questions.

Claude Fable 5.1 scored 75.4% average rubric coverage, 3.3 percentage points ahead of Claude Opus 5.5 on the official leaderboard.

Partial credit does a lot of work here: that score measures the fraction of required criteria satisfied on each question, averaged across questions. Full-credit rates across the 25 models ranged from 12.6% to 26.3%.

Vals AI built Immigration Bench with the Talent Mobility Fund, a program of Renaissance Philanthropy. The methodology separates expert-authored requirements from automated conversation grading:

• Dataset: 519 questions, with 262 held private for testing and 257 reserved for labs to license for their own evaluations.
• Expert review: Each question's rubric was cross-reviewed by a second immigration expert.
• Conversations: Models had up to five exchanges with a simulated person, powered by GPT-6 Luna, answering follow-up questions from supplied background facts.
• Grading: Claude Sonnet 5 checked whether each criterion was satisfied anywhere in the transcript. On validation items where two immigration experts agreed, its judgments matched their consensus 88.6% of the time.

Without web search, average rubric coverage across the 25 models varied sharply by the person asking. The report breaks out seven personas:

• International student: 85.6%.
• International scholar or faculty member: 66.7%.
• University advisor: 48.5%.
• Employer: 64.6%.
• Entrepreneur: 64.2%.
• Self-petitioner: 63.6%.
• Fragile or temporary status: 51.1%.

Fable 5.1 led all seven groups, but reached only 63.9% on questions from people with fragile or temporary status, including Deferred Action for Childhood Arrivals and Temporary Protected Status. University-advisor questions had the longest rubrics, averaging 19.5 criteria each.

The report isolates 65 rubric items citing rules from 2025 or 2026:

• Without search: Models met 7.3% of those items, compared with 64.0% of other criteria.
• Best without search: Fable 5.1 and Claude Opus 5.5 each reached 18.5% on recent-rule items.
• With search: Pooled recent-rule coverage reached about 29%; the best model still remained below 44%.

Employer questions gained 10.4 percentage points with search, the largest improvement by persona. Fable 5.1's overall score gained just 1.2 points.

Models were better at explaining rules than establishing whether those rules applied to the person asking. Rubric-item completion differed by task in the benchmark results:

• Confirming personal circumstances: 43%.
• Explaining the law or recommending a next step: 63%.
• Items warning against a specific mistake: 86%.

Models that asked questions in their first reply earned about four points more on fact-finding criteria. Their overall scores moved by less than one point, and their legal guidance did not improve.

Vals calculated two hindsight ceilings in the full report:

1. Pick the best whole response for each question: About 87% average rubric coverage.
2. Combine the best response to each individual criterion: About 92% coverage, with roughly 62% of questions fully correct.

Even the criterion-by-criterion combination covered only about 29% of recent-policy items. Vals cautions that this upper bound also includes items the AI grader incorrectly marked as satisfied.