Can AI Choose the Right Statistical Test? What 2026 Benchmarks Reveal

A test name should come near the end of the reasoning process, not the beginning. First identify the population and the quantity to estimate: a mean difference, a change within the same unit, an association, a proportion, a rate, a time-to-event contrast, or something else. Then establish how observations entered the dataset, which variables are outcomes and predictors, whether measurements are independent or linked, and what inference the design can support.

Sources:S08S09

The same column types can represent unrelated samples, repeated people, matched facilities, or clustered transactions. A binary outcome may come from a randomized experiment or an observational survey. These designs create different targets and threats to validity. A language model cannot recover facts that were never recorded merely by inspecting a spreadsheet.

Sources:S08S09

Missingness, outliers, censoring, unequal exposure, ties, small cells, repeated looks, and multiple outcomes can change the plan. Checks should combine subject knowledge, design information, plots, and diagnostics appropriate to the model. A preliminary test is not a universal gate that selects the next procedure. NIST guidance treats exploration, assumption checking, fitting, and validation as connected work.

Sources:S08S09S10

Finally, the intended claim must match the design. An association estimated from observational data does not become causal because a regression produced a small p-value. A statistically detectable effect is not automatically large, important, or useful. The ASA’s principles emphasize that a p-value does not state the probability that the null hypothesis is true and that scientific conclusions should not turn on a threshold alone.

Sources:S11

P-Bench and Fisher-R1. Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing is an August 2026 arXiv v1 preprint, not an established peer-reviewed publication. P-Bench contains 425 open-ended tasks divided into 203 Easy and 222 Hard cases. The current public release identifies 405 rows as nonsynthetic and 20 as synthetic, which qualifies the paper’s broader real-world framing. For each task, an agent receives a question and CSV, uses an R sandbox, and returns a method, p-value, and reject-or-fail-to-reject decision.

Sources:S01S02

Raw scores the decision direction. Strict also requires the reported p-value to be close to the canonical value on the paper’s transformed scale. It does not directly score whether the named method is correct. In Table 1, GPT-5.4 recorded 58.3% Raw pass@1 and 30.5% Strict pass@1 on the Hard split under that three-rollout setup. The gap can reveal numerical disagreement, but is not a universal GPT-5.4 score or proof that each Strict failure used the wrong method.

Sources:S01

StatABench. This June 2026 arXiv v1 preprint has two distinct tracks. Stat-Closed contains 404 tasks across 18 topics, including 198 practical tasks that use a 35-function Python toolkit. Stat-Open contains 30 competition-derived projects assessed through a report-scoring process. These are not 434 interchangeable questions. Under the paper’s Stat-Closed setup, GPT-5.1 achieved 68.6% overall accuracy across heterogeneous formats; that number is not a pure test-selection rate.

Sources:S03S04

Configuration also mattered. On the practical subset, DeepSeek-V3 scored 53.54% with the LangChain MCP baseline and 85.35% with CrewAI. That comparison shows that orchestration can alter benchmark results substantially. It does not establish that CrewAI is always superior: the direction and size of changes differed across model-framework pairings.

Sources:S03

StatLLM and StatQA. StatLLM is a peer-reviewed 2026 Scientific Data Data Descriptor covering 65 datasets, 207 specified SAS tasks, and 621 outputs. Nine experts separated correctness, execution, and output quality. With prescribed requests and older named systems, it informs multidimensional evaluation rather than current independent test selection.

Sources:S05S06

StatQA, a peer-reviewed NeurIPS 2024 benchmark, tests constrained selection of applicable methods and columns from supplied metadata and fixed candidates. Its 11,623 examples, including a 1,163-item mini benchmark, are useful background, but do not reproduce open-ended raw-data diagnosis or measure 2026 frontier systems.

Sources:S07

The benchmarks illuminate different stages of statistical work. Their percentages should remain inside their own task, tool, model, and scoring contexts. Because the task sets and scoring methods differ, the percentages are not directly comparable.

Sources:S01S03S05S07

Taken together, the evidence supports a narrow conclusion: statistical automation needs evaluation at more than one layer. A system may identify a familiar procedure but mishandle parameters, execute correct syntax for the wrong estimand, or report a direction that is not supported by its own numerical analysis. Conversely, a lower score may reflect output parsing or an orchestration constraint rather than one indivisible lack of statistical intelligence.

Sources:S01S03S05S07

An AI assistant can be useful before, during, and after computation when its work is treated as a reviewable proposal. Before analysis, it can ask: What is the observational unit? Which units repeat? Was assignment random? Are there clusters, weights, censoring, or multiple outcomes? It can summarize a data dictionary, flag inconsistent codes, and inventory missing information rather than guess.

Sources:S08S09S10

During analysis, it can propose candidate methods with reasons and disqualifying conditions, write commented code, generate plots, check residuals, compare specifications, and trace exclusions. The analyst must still read the code and verify package defaults. P-Bench and StatABench show executable agent workflows; StatLLM shows why execution should be assessed separately from analytical correctness.

Sources:S01S03S05

After analysis, AI can translate output into a draft explanation, build a checklist of caveats, and compare the result with sensitivity analyses. It can ask whether effect estimates and intervals tell a different story from a thresholded p-value. It can also reveal its own uncertainty by naming missing fields, competing interpretations, and the evidence that would change its recommendation.

Sources:S11S12S14

These uses are assistance, not delegation of accountability. The appropriate level of review rises with the consequences of error, novelty of the design, sensitivity of the data, and distance between the benchmark task and the real decision.

Sources:S12S14

Runnable code proves only that a particular instruction was executable in a particular environment. It does not prove that rows are independent, that the target variable means what the prompt claims, that missingness is ignorable, or that the model answers the intended question.

Sources:S05S08S09S10

Common failures include treating repeated observations as independent; selecting a test from keywords; accepting default row deletion; confusing linear with monotonic association; applying a small-cell rule without the sampling design; ignoring clustering; or selecting the smallest result after many analyses. A system can also calculate association correctly and then use unsupported causal language.

Sources:S08S09S10S15S16S17

Interpretation can fail after correct computation. A p-value concerns data-model compatibility under assumptions, not the probability that a claim is true. Failure to reject is not proof of no effect. Significance does not state magnitude, practical value, or replicability. Estimates, intervals, diagnostics, sensitivity checks, and context remain necessary.

Sources:S08S10S11

Automation adds its own risks. A model may invent a variable definition, silently change a hypothesis, select an unavailable package function, or produce a polished explanation that obscures uncertainty. StatABench’s framework-dependent results show that the wrapper and tools are part of the evaluated system. The NIST Generative AI Profile similarly treats confabulation, validation, monitoring, and the human-AI configuration as risk-management concerns.

Sources:S03S14

The following fictional examples are original teaching scenarios. They are not benchmark tasks, field reports, assignment answers, or answer keys. Each shows what an AI assistant might notice, what it could miss, what a human should verify, and which absent facts prevent a final choice.

A reviewable sequence keeps method selection connected to design and reporting: Research question → Study design → Variable types → Data quality → Assumptions → Candidate methods → Analysis → Diagnostic and sensitivity checks → Interpretation → Reporting and limitations.

Sources:S08S09S10S11

Human review can itself fail. A reviewer may lack the needed expertise, trust a polished answer, face incentives to approve quickly, or repeat the same mistaken assumptions. Effective oversight therefore needs named responsibility, adequate time, access to source data and code, escalation paths, and authority to stop the analysis. A second AI output is not independent validation.

Sources:S12S13S14

A useful prompt asks the system to expose reasoning dependencies before it names a procedure. For example: Help me plan an analysis; do not choose a final test until you list the missing design information. Restate my population, observational unit, outcome, comparison, and estimand. Identify dependence, clustering, missingness, outliers, multiplicity, and causal limits. Give candidate methods, assumptions, diagnostics, and conditions that would rule each candidate out. Then provide commented code and a verification checklist. Do not invent variable meanings or data I have not supplied.

Sources:S08S09S10

Attach a de-identified data dictionary and describe how observations were collected. State whether units repeat, whether groups were assigned, what happened before measurement, which exclusions were planned, and which outcome was primary. Ask the tool to separate facts from assumptions and to mark unresolved questions.

Sources:S12S14

Request alternatives. Ask what plot or design fact would change the recommendation, which defaults need review, and how reasonable sensitivity analyses affect the conclusion. For classwork, use the response to understand and document your reasoning; do not request an answer key or present generated work as your own. Follow your institution’s current AI-use and academic-integrity requirements.

Sources:S08S09S10

Better prompting improves transparency, not truth by itself. A detailed prompt cannot restore unrecorded pairing, repair biased sampling, or establish causality. The student or analyst remains responsible for checking the method against course requirements, disciplinary conventions, and the actual data-generating process.

Sources:S12S14

For an organization, a statistical recommendation may influence pricing, hiring, product experiments, quality controls, clinical or safety escalation, and resource allocation. A wrong method can therefore become an operational, compliance, financial, or reputational risk even when the immediate artifact is only a notebook.

Sources:S12S14

NIST AI RMF 1.0 is voluntary, context-sensitive guidance—not a law, certification, or reliability guarantee. Its risk-management approach supports a practical control design: define authorized uses, assign ownership, map affected people and systems, measure performance in the relevant context, and manage identified risks over time.

Sources:S12

DomyClass analysis suggests controls such as classifying analyses by consequence; restricting sensitive data; logging prompts, data, code, packages, and model versions; setting review criteria before results are known; testing failure cases; and creating escalation to a qualified statistician. High-impact decisions require stronger evidence than exploratory work.

Sources:S12S13S14

The human-AI arrangement is itself part of the system. Controls should specify who may accept a method, who can override it, what competence is required, and when automation must stop. A checkbox labeled “human reviewed” is weak evidence unless the reviewer can inspect the inputs, reproduce the analysis, understand uncertainty, and challenge the recommendation. This is the article’s BUSN600 connection: responsible adoption depends on validation, auditability, accountability, and fit-for-purpose risk decisions.

Sources:S13S14