In 2024, a team of 50 researchers across eight laboratories completed a large-scale replication audit of 12 rodent learning studies. The result: only three of the original findings held up under close scrutiny. The project, part of the Many Eyes Replication initiative based at the University of Virginia, cost roughly $300,000 in direct funding from Arnold Ventures and the Templeton Foundation. It stands as one of the most methodologically rigorous tests of rodent behavioral neuroscience to date, and it has sparked debate about what a 25% replication rate actually means.

A Single Replication Project Raises Hard Questions

The project was led by experimental psychologist Brian Nosek, who has spent more than a decade organizing large-scale replication audits. Each of the 12 original studies was retested using preregistered protocols, with original authors consulted on method details to ensure fidelity. Teams of 5–8 labs per study doubled the sample sizes of the originals, and data analysis was conducted blind to condition, with independent code review. The effect-size threshold for a successful replication was set before any data were collected.

Of the 12 studies, only three met the pre-specified criterion: the 95% confidence interval of the replication effect included the original effect size. The nine that failed included studies of object recognition, social learning, and operant timing. Some failures still showed effects in the same direction as the original, but they were smaller and no longer statistically significant. Others produced null results outright.

The project's cost—roughly $25,000 per original study—highlights the resource demands of rigorous replication. That figure includes salaries for research assistants, animal housing, equipment, and data management. It does not include the opportunity cost of the dozens of principal investigators who contributed their time and lab space.

Reactions among researchers have been mixed. Some see the 25% rate as evidence that the field needs to tighten its methods. Others argue that the replication criterion was too strict, and that Bayesian analysis—which can provide evidence for the null—would have changed the count. The debate mirrors similar discussions in psychology and economics, where large-scale replication projects have reported success rates in the 30–50% range.

How Replication Audits Are Designed and Funded

The Many Eyes Replication project was designed to address a persistent criticism of earlier replication efforts: that they lacked the statistical power and methodological rigor to be definitive. Each of the 12 studies was assigned to a team of 5–8 labs, which collectively ran the experiment on a sample size roughly double that of the original. The labs followed a detailed protocol that specified animal strain, age, housing, handling, and testing schedule.

Blind data analysis was enforced: no one involved in data collection saw the results until after the analysis plan was finalized. Independent code review ensured that the statistical scripts were free of errors. The effect-size threshold—whether the replication's 95% confidence interval included the original point estimate—was chosen to balance sensitivity and specificity. Critics note that this criterion can be misleading when the original estimate is imprecise, but it remains a common standard.

Funding for the project came from Arnold Ventures, a philanthropy that supports evidence-based policy, and the Templeton Foundation, which has funded several meta-science initiatives. The total direct cost of roughly $300,000 covered materials, animal per diems, and personnel time. Indirect costs, such as university overhead, were covered by the participating institutions.

The funding model is unusual in that it explicitly pays for replication. Most federal grants in the United States, such as those from the National Institutes of Health (NIH), do not require or budget for independent replication. The Many Eyes project was designed as a demonstration that replication can be done at scale if the resources are allocated. As of early 2025, the project has inspired several follow-up audits in other subfields.

Why Rodent Learning Studies Are a Stress Test

Rodent learning studies are a particularly stringent test of replicability because behavior is exquisitely sensitive to environmental conditions. A difference in the time of day when testing occurs, the type of bedding in the home cage, or the handling technique used by the experimenter can shift performance. Even the strain of the animal matters: a study that works in Long-Evans rats may fail in Sprague-Dawley rats.

The 12 original studies used a variety of tasks: fear conditioning, spatial memory in the Morris water maze, object recognition, operant conditioning, and social learning. These tasks are staples of behavioral neuroscience, but they are also known to produce variable results across labs. One replication in the project failed because of a lighting difference: the original study used a dim red light during testing, while the replication labs used standard white light, which altered the animals' anxiety levels.

Efforts to standardize procedures—known as standard operating procedures (SOPs)—can reduce but not eliminate this noise. The Many Eyes project required all labs to follow a detailed SOP, but minor deviations were inevitable. For example, one lab's water maze was slightly larger than specified, and another lab used a different brand of food reward. These small differences can accumulate and affect the outcome.

Some researchers argue that rodent learning studies are too variable to be a fair test of replicability. Others counter that if a finding cannot survive this level of natural variation, it may not be robust enough to serve as a foundation for further research. The tension between standardization and ecological validity is a central theme in the ongoing debate about replication in animal behavior.

The Three Studies That Held and the Nine That Did Not

The three studies that successfully replicated came from well-established laboratories and used tasks with a long history of reliable effects. One was a study of context fear conditioning from the Rudy lab, which showed that rats freeze more when placed in a chamber where they had previously received a mild foot shock. The replication produced a nearly identical effect size, with overlapping confidence intervals.

Another was a spatial memory study from the Morris lab, using the water maze task in which rats learn to locate a hidden platform. The original finding—that rats with hippocampal lesions are impaired—was confirmed. The third was a cued fear extinction study from the Quirk lab, which found that extinction training reduces fear responses to a tone previously paired with shock. All three replications used sample sizes roughly double the originals and achieved high statistical power.

The nine failures included studies of object recognition, where rats spend more time exploring a novel object than a familiar one; social learning, where rats learn from conspecifics; and operant timing, where rats press a lever after a fixed interval. In several cases, the replication produced an effect in the same direction but smaller—sometimes half the original magnitude. In others, the effect was near zero.

One pattern emerged: the nine studies that failed had smaller original effect sizes on average than the three that held. The median original effect size for the failures was roughly Cohen's d = 0.6, compared to d = 1.2 for the successes. This suggests that some of the original findings may have been overestimates due to small samples and publication bias. The project's results are consistent with the idea that larger effects are more likely to replicate.

Methodological Choices That Produced the 25% Rate

The 25% replication rate is not a simple number; it depends on several methodological decisions made by the project team. The primary criterion—whether the 95% confidence interval of the replication included the original effect size—is a standard but conservative choice. It requires the replication to be consistent with the original estimate, not just statistically significant. If the criterion had been simply whether the replication produced a statistically significant effect in the same direction, the success rate would have been higher, roughly 50%.

Using Bayesian analysis would also change the count. A Bayesian approach can quantify evidence for the null hypothesis, which some failures might have supported. The project team reported Bayesian analyses in supplementary materials, but the main results used the frequentist criterion. This choice has been criticized by some statisticians who argue that Bayesian methods are more appropriate for replication because they can distinguish between "no evidence" and "evidence of no effect."

Pre-registration prevented p-hacking and selective reporting, which are known to inflate false-positive rates. Each lab committed to a data-collection plan before seeing any results. This eliminated the possibility of running more animals until a significant result emerged or of dropping outliers to improve the p-value. However, pre-registration also meant that labs could not adapt to unexpected problems, such as equipment malfunctions or animal illness, which may have reduced the signal in some replications.

The debate over direct versus conceptual replications also colors the interpretation. Direct replications attempt to copy the original methods as closely as possible, while conceptual replications test the same hypothesis with different methods. The Many Eyes project used direct replications, but some critics argue that conceptual replications are more informative because they test the generality of the finding. Others reply that direct replications are the first step in diagnosing why a finding may not hold up.

What Funding Agencies Are Learning From This

Funding agencies have taken note of the Many Eyes results. The NIH now requires replication plans for some R01 grants, particularly in areas where prior findings have been difficult to reproduce. The requirement is not a mandate for independent replication but rather a plan for how the grantee will internally verify key results. Some institutes, such as the National Institute of Mental Health, have gone further by funding multi-lab replication projects.

The National Science Foundation's Social and Economic Sciences division has launched a program to fund audit studies, in which independent teams retest published findings. The program is small—roughly $2 million per year—but it has funded replication audits in economics, sociology, and political science. The European Research Council is piloting a replication voucher system, where grantees can apply for supplementary funding to have their key results independently replicated.

Private foundations are also shifting their approach. Arnold Ventures, which funded the Many Eyes project, has moved toward multi-lab consortium grants that build replication into the research design from the start. The Templeton Foundation has funded several meta-science projects that examine the factors that make findings more or less replicable. These foundations see replication as a way to increase the return on their research investments by ensuring that funded findings are robust.

The cost of a full audit—roughly $25,000 per study—is modest compared to the cost of building on irreproducible results. A single clinical trial based on a false finding can cost millions and lead to wasted resources. The Many Eyes project suggests that systematic replication at scale is feasible if agencies allocate a small fraction of their budgets to it. As of late 2024, several agencies are considering dedicating 1–2% of their research portfolios to replication.

Practical Takeaways for Researchers and Editors

For individual researchers, the clearest lesson is to use registered reports for confirmatory work. Registered reports involve peer review of the study design before data collection, which eliminates publication bias and encourages rigorous methods. Several journals in neuroscience now offer registered reports, and their use is growing. Researchers can also share raw data and analysis code at submission, making it easier for others to verify the results.

Journal editors have a role to play by enforcing method checklists. Many journals require authors to state sample sizes, exclusion criteria, and randomization procedures, but enforcement is often lax. A 2023 audit of 50 neuroscience journals found that fewer than half verified that authors had followed the checklist. Stronger enforcement could reduce the number of underpowered studies and false positives.

Training in meta-analysis and power analysis is essential for graduate students and early-career researchers. Many researchers are taught how to run a t-test but not how to estimate the sample size needed to detect a realistic effect. Workshops on reproducible research practices, offered by organizations such as the Center for Open Science, can fill this gap. Some universities now require a course in research methods and statistics for all doctoral students in the life sciences.

Finally, replication rates are a metric, not a verdict on a field. A low replication rate does not mean that all findings in a field are false; it means that the evidence base is weaker than previously thought. The Many Eyes project has prompted self-reflection among rodent learning researchers, and several labs have begun to systematically replicate their own key findings before publishing them. This kind of internal replication culture may be the most durable outcome of the audit.