Last week, Jack Clark, co-founder of the frontier AI lab Anthropic, and Marina Favaro, head of Anthropic’s in-house think tank, published an essay that called for slowing down the development of artificial intelligence. The basis of their concern was that AI companies could be on the verge of creating “recursive self-improving” models, meaning that the AI systems can develop new models without human input.
To understand why this is both meaningful and worrisome to Anthropic, consider how widely used large language models—like Anthropic’s Claude or OpenAI’s ChatGPT—are created. While portions of the coding are delegated to computer-generated “agents” that can run the code themselves, human engineers are still in the loop. Humans initiate the coding processes, design the programs and conduct quality assurance. But Anthropic postulates that software development conducted without any human input is approaching. And humanity is not ready.
It’s one thing for a tech company to hype the next evolution in software development. Whether that hype should be believed is another matter. But this announcement seems different: It was a warning, and a call for action. Fully recursive self-improvement, once achieved, “might increase the risks of humans losing control over AI systems,” Clark and Favaro wrote. The solution was to find ways for human developers to cooperate to “secure them, monitor them, and shape their behavior.” Until such cooperative efforts could be achieved, they called for slowing or even temporarily pausing frontier AI development.