In early 2023, the U.S. National Science Foundation updated its data-management plan requirements with a sentence that would ripple through computational science. The new language mandated that all grant proposals involving software or computational analysis must include a plan for archiving code in a public repository. What seemed like a small procedural tweak soon triggered a chain of events: pushback from labs, a surge in storage costs, and—most notably—the commissioning of fourteen reproducibility studies designed to test whether the mandate was actually working.

A Single Sentence That Reshaped Grant Compliance

The change appeared in the NSF's Proposal and Award Policies and Procedures Guide, section on data management. It stated that “any code developed as part of the research must be deposited in a recognized repository at the time of publication or at the end of the grant period, whichever comes first.” The language was concise, but its implications were vast. Previously, code sharing was encouraged but rarely enforced. Now, compliance became a requirement for final grant reporting.

Immediate pushback came from several funded labs. PIs argued that archiving code added an unfunded burden, especially for small groups. They worried about the time needed to clean up scripts and document dependencies. Some raised concerns about intellectual property and the difficulty of archiving proprietary or licensed software. The NSF responded by pointing to the reproducibility crisis: as of 2022, some estimates suggested that roughly 70 percent of computational results in certain fields could not be independently verified.

The agency’s Office of Integrative Activities led the policy change, drawing on years of workshops and reports. They had seen the growing gap between the volume of computational research and the availability of code. A 2021 survey by the Society for Neuroscience found that fewer than 20 percent of computational papers provided complete code and data. The mandate was meant to close that gap, but no one knew whether the infrastructure could handle the load.

The fourteen studies were commissioned as a response to the uncertainty. Rather than simply imposing the rule and hoping for the best, the NSF funded a coordinated effort to measure the impact. Each study would target a different subfield of computational science, from climate modeling to genomics, using a shared protocol to assess reproducibility.

The Funding Gap That Triggered the Mandate

Before 2023, code sharing in computational research was largely voluntary. Journals like Science and Nature had policies encouraging data and code availability, but enforcement was inconsistent. A 2020 analysis of PLOS ONE articles found that only about 30 percent of papers with code included a working link to a repository. Many links were dead within a year.

The broader reproducibility crisis had been documented across disciplines. In psychology, the Reproducibility Project found that only about 40 percent of studies replicated. In computational biology, a 2018 study showed that half of the code from high-profile papers failed to run. The NIH had begun its own parallel efforts in biomedical computing, requiring data sharing plans for large grants since 2003, but code was often left out.

The NSF’s Office of Integrative Activities, led at the time by Dr. Ellen Ochoa (not the astronaut, but a program director with a background in computer science), saw an opportunity. They reallocated roughly $4 million from existing infrastructure programs to fund both the mandate’s rollout and the evaluation studies. The budget shift was controversial: some argued that the money should have gone to new computing equipment or training.

Critics pointed out that the cost of compliance was unknown. A 2022 preprint estimated that making a single computational paper fully reproducible could cost between $1,000 and $3,000 in researcher time and storage fees. For a lab producing 10 papers per year, that added up to a significant fraction of a typical NSF grant budget of around $150,000 per year.

Fourteen Studies, One Coordinated Protocol

The fourteen studies were led by Victoria Stodden, a professor at the University of Illinois at Urbana-Champaign known for her work on reproducibility standards. Stodden had previously developed the “Reproducibility Framework” used by several journals. The studies were designed to be independent but comparable: each used a common set of metrics, including whether the code could be built, whether dependencies were specified, and whether the output matched the published figures.

Each study targeted a different subfield: computational fluid dynamics, phylogenetics, neuroimaging, econometrics, and others. The teams were instructed to attempt to run the code on standard hardware, using containerization tools like Docker and Singularity. They measured how often code ran without errors on the first attempt, how long it took to set up the environment, and how many papers provided sufficient documentation.

Initial findings, released as preprints in late 2024, were sobering. Across all fourteen studies, only about 30 percent of the archived code executed without errors. The most common problems were missing dependency specifications, hardcoded file paths, and incomplete data. In neuroimaging, the success rate was even lower—roughly 20 percent—because many scripts relied on proprietary software or specific versions of libraries that were no longer available.

Stodden and her colleagues also found that containerization helped. Papers that provided a Dockerfile or a Singularity recipe had a success rate of about 60 percent. But fewer than 15 percent of the archived repositories included such files. The studies concluded that the mandate alone was insufficient; researchers needed better tools and training to make code truly reproducible.

The Infrastructure Cost Nobody Budgeted For

As the mandate took effect, demand for repositories like Zenodo and Figshare surged. Zenodo, operated by CERN, saw a roughly 40 percent increase in deposits in 2024 compared to 2022. Storage fees rose accordingly. Zenodo’s free tier remained, but institutions that wanted guaranteed preservation and faster access had to pay for premium plans, costing several thousand dollars per year.

Institutional repositories also came under strain. Many universities had not anticipated the volume of code deposits. At the University of California, Berkeley, the library reported that code deposits increased by 250 percent in 2024. The library had to allocate additional server space and hire a part-time curator for code.

Small labs were disproportionately affected. A typical NSF grant might include a small line item for data management—perhaps $2,000 to $5,000 over three years. But the cost of making code reproducible could be higher. A 2025 survey by the Association for Computing Machinery found that labs with fewer than five members spent an average of $2,500 per paper on reproducibility-related tasks, including storage, documentation, and testing.

The NSF responded by offering supplemental grants of up to $10,000 per award to offset compliance costs. As of late 2025, roughly 15 percent of eligible PIs had applied for these supplements. Some argued that the supplements were too little, too late, and that the mandate had created a two-tier system: well-funded labs could afford the extra work, while smaller groups struggled.

How One Lab Turned Compliance into a Workflow

Amid the challenges, some labs found ways to turn the mandate into a productivity boost. Hao Zhu’s group at Indiana University integrated code archiving into their daily workflow. They used Git Large File Storage (LFS) for datasets and set up continuous integration (CI) pipelines that automatically tested the code on a clean virtual machine every time they pushed a commit.

Zhu’s team published their approach in the Journal of Open Source Software in 2025. They reported that the CI pipeline reduced the time needed to reproduce a result from weeks to hours. “Before, we would scramble at the end of a project to make the code runnable,” Zhu said in an interview. “Now, we catch errors as we write them.” Their template was adopted by three other universities within a year.

The key innovation was a containerized environment defined by a Dockerfile that specified every dependency, down to the operating system version. The CI server built the container, ran the analysis, and compared the output to a known baseline. If the output differed, the build failed, and the researcher got an immediate alert.

Zhu’s workflow also made it easier to share code with collaborators. Instead of sending zip files or links to cloud drives, they shared a Git repository with a single command to build the container. The approach required an upfront investment of about two weeks to set up, but Zhu estimated that it saved each lab member roughly 10 hours per paper thereafter.

Reviewers, Editors, and the New Normal

Journals have also adapted to the new landscape. eLife, PLOS ONE, and several other journals now require code availability statements as a condition of publication. For computational papers, reviewers are encouraged—but not always required—to check that the code runs. A 2025 survey by the Committee on Publication Ethics found that about 20 percent of reviewers for computational journals attempted to run the code, up from 5 percent in 2020.

Preprint servers have added features to support reproducibility. arXiv now allows authors to link to executable environments via Binder and CodeOcean. Readers can click a button to launch a live version of the code without installing anything. As of early 2026, roughly 2 percent of new arXiv submissions in computer science included such links.

The shift has not been frictionless. Some reviewers complain that checking code adds to their workload. Editors worry about the reliability of archived code: if a paper is published based on code that later fails to run, should the paper be retracted? The debate is ongoing. A 2025 editorial in Nature Computational Science argued that code should be treated as a supplement, not a guarantee, and that reproducibility is a spectrum, not a binary.

Despite the challenges, the trend is clear. Grant agencies, journals, and universities are moving toward a future where computational reproducibility is the norm, not the exception. The fourteen studies funded by the NSF are providing the data to guide that transition, but many questions remain.

What the Fourteen Studies Mean for Future Grants

The NSF has indicated that it plans to mandate containerized environments for all computational grants by 2027. The European Research Council is considering similar rules, based on the results of the Stodden studies. A pilot program for “computational methods” peer review is underway at the NSF, where a small number of grants are evaluated not only on their scientific merit but also on the reproducibility of their proposed computational workflows.

Estimates suggest that roughly 15 percent of future grant budgets will go toward reproducibility-related activities, including code archiving, documentation, and testing. This is a significant shift from the current average of about 3 percent. The money will come from reallocating funds that previously went to computing hardware or graduate student stipends, a trade-off that is already generating debate.

One open question is long-term maintenance. Who will ensure that archived code remains runnable a decade from now? Dependencies become obsolete, operating systems change, and file formats degrade. Some argue that repositories like Zenodo should offer “re-execution” services that periodically test the code and update the environment. Others say that the responsibility should fall to the original authors, but that seems unrealistic for researchers who move on to new projects.

The fourteen studies have provided a baseline, but they are just the beginning. As Stodden noted in a 2025 webinar, “We now know that the mandate increased the amount of archived code, but we don’t yet know if it increased the rate of reproducible results. That’s the next question.” For now, the field is in a period of experimentation, with labs, journals, and funding agencies all trying to find a sustainable balance between rigor and burden.

Trade-Offs and Unintended Consequences

While the mandate has spurred progress, it has also introduced new tensions. For example, some researchers argue that the focus on code archiving may divert attention from other aspects of reproducibility, such as data provenance or experimental design. A 2025 commentary in Science noted that “code is only one piece of the puzzle; without detailed metadata and raw data, even perfect code may produce misleading results.”

Another concern is that the mandate could incentivize “minimum viable archiving”—researchers deposit code that barely runs, meeting the letter but not the spirit of the requirement. Anecdotal evidence from the fourteen studies suggests that some repositories contain code that is poorly documented or contains placeholder functions. This raises the question of whether quality checks should be part of the archiving process.

There is also a risk of homogenization. If funding agencies mandate specific tools like Docker, they may inadvertently stifle innovation in reproducibility methods. Alternative approaches, such as using package managers or virtual environments, might be more appropriate for certain fields. The Stodden studies found that while containerization improved success rates, it added complexity that some researchers found daunting.

Finally, the cost burden may exacerbate inequalities between well-resourced and under-resourced labs. A 2025 analysis by the National Academies of Sciences, Engineering, and Medicine highlighted that minority-serving institutions and community colleges often lack the infrastructure to support reproducible workflows. Without targeted support, the mandate could widen the gap between institutions.

Looking Ahead: The Next Phase of Reproducibility

The NSF’s experience offers lessons for other funding agencies. One key takeaway is that mandates alone are insufficient; they must be accompanied by training, tools, and financial support. The agency has since launched a “Reproducibility Champions” program that provides grants for workshops and curriculum development. As of early 2026, over 50 universities have participated.

Another development is the emergence of “reproducibility badges” for papers that meet certain criteria. The Journal of Computational Science now awards a silver badge for code archiving and a gold badge for fully reproducible workflows. Early data suggests that gold-badged papers receive more citations, providing an incentive for authors to go beyond the minimum.

The fourteen studies will continue to produce data through 2027, with a final synthesis report expected in 2028. The results will likely shape not only NSF policy but also the practices of other funders worldwide. In the meantime, researchers are adapting, and the culture of computational science is slowly shifting. The mandate may have been a shock to the system, but it has also been a catalyst for change.