loading
loading
Vals AI's Immigration Bench Exposes Claude Fable 5.1's Legal Blind Spots Vals AI tested 25 frontier models on 262 expert-written U.S. immigration questions and found even the best model leaves out one in four essentials. - Vals AI and Talent Mobility Fund launched Immigration Bench, 519 expert-written U.S. immigration questions across 7 personas. - Top model Claude Fable 5.1 met only 75.4% of rubric items, fully correct 26.3% of the time. - Models met just 7.3% of items on 2025 and 2026 rules versus 64% on established law. - Rubric items requiring applying law to facts met 43%, versus 63% for explaining rules. - Web search added 10.4 points on Employer questions but pushed Claude conversations to 70 minutes each. - Combining best answers across all 25 models would score 92%, suggesting ensemble gains. Vals AI and the Talent Mobility Fund have released Immigration Bench, a test of 25 frontier models using 262 realistic U.S. immigration questions written by practicing attorneys. The benchmark covers visas, work authorization, legal status, employer obligations, and other issues where an inaccurate answer can affect a person’s job or ability to remain in the country. Claude Fable 5.1 led the no-search leaderboard, satisfying 75.4% of expert rubric requirements on average. It satisfied every requirement on 26.3% of questions. The first figure measures item-level coverage; the second measures complete answers with no essential element missing. The full report cites Pew Research Center data indicating that about half of U.S. adults use AI chatbots. Immigration queries carry unusual risk because a missed deadline, incorrect classification, or outdated policy can jeopardize employment, status, or a pending application. Inside the five-turn test Immigration specialists drafted 519 questions reflecting the ambiguity and detail encountered in practice. Each case adopts a specific persona, including an international student, employer, entrepreneur, DACA recipient, university adviser, or person with Temporary Protected Status. Published results use a private 262-question test set to reduce contamination and benchmark gaming. Each question carries an expert-written rubric that identifies the facts, legal rules, warnings, and recommended steps required for a complete answer. This approach distinguishes an answer that sounds plausible from one that covers the details an attorney considers essential. Evaluation unfolds over as many as five conversational turns, with separate systems assigned to answer the question, simulate the user, and grade the resulting transcript: - Tested model: Answers the initial question and may request additional facts. - Simulated user: GPT-6 Luna responds to follow-up questions using only information available in the assigned persona. - Grader: Claude Sonnet 5 checks every rubric item against the full transcript. On items where two immigration experts agreed, its verdict matched theirs 88.6% of the time, the highest agreement among the graders tested. Fact-finding drags down legal answers Models handled general legal information more reliably than case-specific analysis. They satisfied 63% of rubric items requiring an explanation or recommended step and 86% of items requiring a warning against a specified mistake. Coverage fell to 43% when an item required confirming a fact about the user’s circumstances. Responses that asked clarifying questions in the first turn earned about four additional percentage points on fact-finding items for that question. Their overall score rose by less than one point, and the quality of the subsequent legal guidance remained unchanged. The result suggests that effective intake requires more than prompting a model to ask additional questions; the system must also apply the answers correctly. New policy falls through the cutoff Immigration policy changes frequently through executive orders, agency memoranda, regulations, and administrative decisions. Model weights have fixed training cutoffs, leaving recent rules poorly represented unless a system retrieves current, authoritative material. The benchmark includes 65 rubric items tied to rules from 2025 or 2026. Across all 25 models without search, those items were satisfied 7.3% of the time, compared with 64% for the remaining items. Claude Fable 5.1 reached 18.5% on the recent-rule subset, while a hypothetical oracle selecting the best no-search response from all 25 models reached about 29%. Search-enabled runs also satisfied about 29% of recent-rule items across models, and the strongest individual result reached roughly 43%. General web access therefore improved coverage without resolving the underlying policy gap. The report gives three examples: - Remote I-9 verification: No model surfaced a March 2026 ICE change without search; 9 of 25 found it with search. - TPS travel: No model mentioned an August 2026 Board of Immigration Appeals decision without search; 19 of 25 found it with search. - F-1 Duration of Status: Only 3 of 25 models correctly distinguished admissions before and after the July 2026 DHS rule, even with search enabled. Complex cases lose the most coverage Persona-level results varied sharply. International Student questions averaged 85.6%, while University Adviser questions averaged 48.5%. The adviser cases also had the longest rubrics, with 19.5 required items per question, indicating that models lose coverage as the number of relevant facts and obligations grows. Questions in the Fragile/Temporary Status category, including DACA and TPS cases, averaged 51.1%. The best model reached 63.9% for this group, leaving more than one-third of required information unaddressed even in the strongest result. The same pattern appeared across immigration pathways. Models satisfied 76% of rubric items concerning Optional Practical Training and STEM OPT. Coverage fell to between 57% and 60% for longer-term employment and talent pathways such as National Interest Waivers, PERM labor certification, O-1 visas, and EB-5 visas. Search buys uneven gains and long waits Web search improved Employer questions by an average of 10.4 percentage points and University Adviser questions by 6 points. International Student questions gained 1.6 points, showing that retrieval benefits depend heavily on the type of case. Search volume failed to predict improvement. GPT-6 Luna conducted more than 100 searches per conversation and gained 1.8 points. DeepSeek V4 Pro averaged about 10 searches and gained 6.5 points. Retrieval quality, source selection, and the model’s ability to apply retrieved material mattered more than query count. Search also increased latency substantially. Claude Opus 5.5 took a median of 70 minutes per conversation with search, up from 13 minutes without it. Claude Sonnet 5.5 rose from 11 minutes to 63 minutes. Those response times complicate synchronous legal-assistance products and require explicit timeout, fallback, and escalation behavior. An oracle exposes model complementarity A hypothetical oracle that selected the best response to each rubric item across all 25 models would score about 92%, compared with 75.4% for Claude Fable 5.1. It would also produce a fully correct answer on about 62% of questions, more than double Claude Fable 5.1’s 26.3% rate. The oracle result establishes an upper bound rather than a deployable architecture. A production ensemble would need to identify the strongest answer, reconcile contradictions, preserve citations, and control cost and latency. Recent-policy coverage would remain constrained by access to current primary sources. Lower prices bring measurable gaps Lower-priced models approached the leader on cost more closely than on coverage. The report lists the following no-search results: Actual deployment costs will vary with token usage, retries, search calls, and provider pricing. The lowest-scoring model trailed the leader by 19.7 percentage points, a material difference for free or low-cost services likely to handle questions from people without ready access to counsel. Read the leaderboard within its scope The benchmark directly measures rubric coverage in simulated conversations; client outcomes, filing accuracy, and courtroom performance remain outside its scope. The simulated user also answers consistently from a fixed persona, while real users may omit facts, misunderstand terminology, or provide conflicting information. The automated grader achieved 88.6% agreement on rubric items where two experts shared the same verdict, leaving some measurement error. The private test set reduces contamination and also limits independent review of individual cases. Small differences between models therefore deserve caution, especially because longer answers can satisfy more rubric items through added coverage. Production systems need stronger guardrails - Connect retrieval to maintained policy sources. Use agency feeds, regulations, memoranda, and administrative decisions with effective dates and jurisdiction metadata. General web search produced inconsistent coverage of recent rules. - Separate intake from legal guidance. Collect required facts in structured fields, validate conflicting answers, and prevent the system from issuing a recommendation until essential information is present. - Display sources and freshness. Every policy-dependent claim should include its authority, publication date, effective date, and retrieval time so users and reviewers can identify stale guidance. - Escalate high-risk categories. DACA, TPS, employer compliance, and long-term employment pathways showed lower coverage and warrant review by a qualified professional. - Evaluate more than aggregate accuracy. Track full-answer accuracy, unsupported claims, missed deadlines, citation quality, latency, and performance by persona. Treat leaderboard differences below two percentage points cautiously. - Validate routing and ensembles end to end. The oracle score supports experiments with model routing, while production testing must account for conflicts, synthesis errors, cost, and response time. Immigration practices can use language models for research, issue spotting, and first drafts when a qualified professional reviews the output. Tailored legal advice still depends on complete facts, current policy, and professional judgment. AI developers need direct access to authoritative immigration materials and ongoing input from practitioners. Larger pretraining runs cannot supply agency memoranda, adjudication trends, or newly effective rules that remain absent from the accessible corpus. Reliable systems will require maintained policy data, explicit uncertainty controls, and human escalation paths.
document fetched 2026-10-29 03:02