AI has changed the landscape for research software. It’s now trivial
to write more complex “throwaway” scripts — the kind that read some
data, run an analysis, and spit out a result — doing more of that
process “from scratch,” and with fewer dependencies than before. A
robustness check that used to mean pulling in three half-remembered
packages and an afternoon of pip install errors can now be
sketched, debugged, and run in the time it takes to describe what you
want.
This makes it easier than ever for people (willing to use AI, mind) to do computationally-minded research, even people who’ve never really learned to program. The bottleneck used to be knowing enough software engineering to get from “I have an idea” to “I have working code that answers it.” That bottleneck is eroding fast and unevenly.
The unit economics of code have changed
Fundamentally, this reflects a change in what I’d call the unit economics of software. Writing code used to be expensive relative to the value of any one script; because it was difficult to get the code written correctly and an increasing amount of code was needing to be written, there were increasing incentives provided to scientists to make their code reusable: we started to build libraries, write documentation, test edge cases, and treat code as an asset worth maintaining. I’ve written about this elsewhere as productionised science; at what point does every paper have to be a software package? How long do we have to maintain our paper-packages? Forever?
This internal contradiction was building and building within academic practice over the 2010s and early 2020s. Maintenance budens grow increasingly expensive, as academic software, never finished, grew in scope and userbase. Indeed, the Bridges and Roads report documented the growth of these maintenance costs to the point of burnout and unsustainability across the open source community. We start to see “sustainability” emphasized in the assessment of software project funding.
Now, Claude Code and Codex completely reframed the incentive structure for code in academia. Code is increasingly cheap to produce now because of it. When something is cheap to produce, we’re less willing to pay to maintain it — it’s easier to regenerate it from scratch than to understand, patch, and preserve an old version. Throwaway code is being treated as throwaway, because for the first time, that’s actually the cheaper option.
What “reproduction” means when scripts are free
So, what happens to reproducibility, then? When people think of “reproduction” as the ability to re-generate the results from a given script and dataset, AI makes this easier than ever. It’s cheaper to write the script that produces the numbers, and cheaper still to re-run it, or to have someone (or something) else regenerate an equivalent one from the same description. Indeed, why care about the exact script at all? Describe the result to the AI and see if it can achieve it.
But, in an age of prompted software, these scripts are still important. They codify what, exactly, the fact claim of a paper is, and how it was obtained. Multiverse analysis suggests that even very slight changes to designs can lead to considerably different conclusions, and the sensitivity to these changes does not necessarily imply the invalidity of the result!
Indeed, the one-off scripts can become increasingly baroque, isolated, and idiosyncratic, unless we start pooling our resources into libraries that are correct, validated, and documented. A hundred analysts each generating their own bespoke pipeline for a problem that’s fundamentally the same one, none of them checked against each other, isn’t really reproducibility in any meaningful sense — it’s a thousand unaudited implementations of the same idea, some of them wrong in ways nobody will ever notice, because nobody else runs, or even reads, that particular script.
Why libraries still matter
So, this returns us right back to libraries being important. But, what are they really important for if the code itself is now less valuable? Indeed, the value of libraries now that GenAI has completely changed the economics of open source software comes from a different place; it’s not really really about what the code does or doesn’t do computationally — any sufficiently motivated person, human or AI, can write code that produces a number. Instead, it’s about correctness, convention, and community. Correctness for the library suggests that it works, in the sense that it generates scientifically-valid numbers from input. Convention for the library suggests that one does not necessarily have to defend it; inclusion of a technique, approach, or method into an open source library gives it a “thingness” that a paper or replication code does not itself; it reifies patterns of thought (for good or for ill). Community refers to the value that a library includes authors who care about a particular topic enough to work on it.
Indeed, these three aspects reinforce one another, and built upwards naturally. Libraries as communities provide a venue in which a group of scholars come together to collaborate on defining conventions and, ultimately, generate correct techniques. Conventions emerge from the community through discussion, collaboration, and cooperation; correctness emerges from convention as it attains epistemic solidity—repeat validation of the utility of the approach across many empirical or theoretical contexts. Starting with correctness (and placing it at the top of the hierarchy) might suggest that we’re placing it in a privileged position. Yet, I think it’s actually most valuable from below; the scientific community is itself first a community, and then one interested in science. Code frames the analytical process, and does so by emerging directly from individual scholars’ practices. Indeed, we know this because different libraries structure analysis differently, and make certain questions more immediate, more visible, or more natural to ask than others. Indeed, this is intentional: it reflects the body of knowledge embedded in the people who write, edit, and maintain the code and its documentation — the developers of the library.
That framing, and the convening that occurs to generate it, is really the value of the library itself. The code is now a nucleating point, a seed for knowledge to crystallize. This value has now totally swamped the value of whatever numbers library code is able to generate. Correctness is still expensive, in the sense that it requires us to critically apply a technique across a wide and complicated variety of scenarios, in research contexts that were never envisioned directly by the software’s original authors. And convention and community are both more expensive still, in the sense that the developers of a library have spent real time working with tools, methods, data, theory, and other people, to cultivate a distinct perspective.1
The effects of GenAI on libraries themselves
Thus, I think we’ll see libraries maintaining smaller codebases as their core, while actually cultivating a much larger set of how-tos, tutorials, and worked examples — a way of framing that hard-won experience and communicating it to a wider audience, robo or no. This will reflect the body of work that the community engages in. From this work, conventions emerge, and a much smaller set of “correct” code implementations will ultimately bear the cost of being maintained. This change will magnify the importance of, say, showing up to developer/community calls, while simultaneously reducing the importance of actually contributing lines of code.
If that’s right, the job of a library author is shifting a little: less about writing the one correct implementation, and more about writing the thing — human-readable, AI-readable — that teaches whoever’s asking how to think about the problem in the first place. Indeed, some would even argue that the code was never really the point of the library. It was always the evidence of the thinking, intent, and understanding behind it. Unfortunately, that’s also the part that’s truly getting harder to verify.