Article 34 | deewilsoncon

What is Artificial General Intelligence?

On September 11, I sent out a review of a chapter of Adam Becker’s book, More, Everything, Forever, about Ray Kurzweil’s vision of an approaching transformation of human (and cosmic) history titled “The Singularity is Near.” Kurzweil imagines that super intelligent AI will escape not only human control but all limitations that confine its development to Earth, our solar system and galaxy, with the goal of using all available energy in the entire universe to increase its intelligence and computing power. I described Kurzweil as an AI prophet and the Singularity as his revelation.

In this article, I return to Becker’s book to discuss the meaning of Artificial General Intelligence (AGI), the pursuit of which is a common goal and expectation of developers of advanced AI systems. I also make use of Karen Hao’s discussion of AGI in her book, Empire of AI. (2025).

Both Becker and Hao comment on the definitional ambiguity of AGI. Becker questions whether, independent of AI development, there is any meaning to the concept of “general intelligence”. He asserts: “the actual existence of some measure of overall intelligence, ... or just g for short, is debatable at best.” And: “Without a concrete notion of general intelligence, it’s not clear what would constitute an intelligence explosion,” nor what it would mean for one computer to be twice as intelligent as another. Defining AGI in the first place looks shaky too. But even if we accept a vague provisional definition for AGI ... “An artificial machine that can do everything a typical human adult can do,” it’s still not obvious that a modern computer would be capable of running an AGI.”

Hao comments “that the central challenge of working at Open AI was that no one really knew what AGI would look like.” Perhaps as AI systems increased their computing power AGI would reveal itself, Hao states in her comments on one Open AI developer’s thinking. Hao asserts that due to uncertainty about the definition of “general intelligence,” “the field of AI has gravitated toward measuring its progress against human capabilities.” By this measure, advanced AI already exceeds human intelligence at chess and other difficult cognitive games and at mathematics, but has not yet achieved advanced expertise in many other fields. Some commentators describe the progress of AI as “jagged,” rather than general and uniform. AI systems are better than humans at drawing out the logical implications of rules. axioms, established medical science, but are not as impressive (yet) in making discoveries that require creative inspiration, exploration and verification.

Much of the fascination and anxiety of AI developers regarding AGI concerns AI agency and autonomy rather than computing power per se. The expectation (and fear) of many AI developers is that at some point AI systems will become self directing in a way that is not reversible. Becker states that one of the tasks of “strong AI” “would be the design of the next AGI.” And: “Say that first AGI, built by humans, would take thirty years to create. Building an even better, smarter one, might take those same humans more than sixty years. But with an AGI in hand, it would be easy to speed that up: just ask the AGI to design it and give it it immense computational resources. Then that first AGI could do work that would take humans sixty years in just a year, or even a few a few days.” In recent discussions of the Hugging Face hack, the capacity of AI systems to guide their own development has been referred to as “recursive self improvement.”

The idea that AGI would quickly lead to superhuman intelligence has been widespread among AI developers for this reason: once AI becomes self directing they believe AI will transcend human limits and find the means to vastly increase its power. Discussions of AGI have become a veiled way of talking about ceding power to determine the human future to AI. From this perspective, it’s only a matter of time until humans are the servants of AI “agents” rather than vice versa. It is hard to miss that this vision of an AI controlled future sounds like a projection of the world tech lords would like for themselves and their endeavors.

Both Becker and Hao are skeptical about AGI, both as a possibility and an ideal. Becker asserts that the capacity of AI developers to increase the number of transistors on chips is approaching the limits of what’s physically possible. One expert informed Becker that “The rate at which transistor lateral dimensions continue to scale down with each new generation of manufacturing technology will slow down or even stop, due to physical limits and or financial considerations. “ Becker states: the physical limits ... arise somewhere about 0.1 to 1 nanometer because atoms of silicon are about 0.2 nanometers across, and you can’t build a transistor out of less than a single atom. This limit isn’t possible to break, and we’re on the verge of reaching it ...” and: “The exponential trend is over.”

However, neither Becker or Hao discuss the possibility that AI agents could become vastly more powerful by working together, as occurred in the Hugging Face hack in which “swarms” of AI agents found one another and divided up tasks to achieve a common goal. At some point in the history of humanity, increases in knowledge and improvements in technology became driven by cultural evolution rather than biological evolution. If or when advanced AI systems become able to develop stable social relationships among themselves, it’s impossible to anticipate what might occur.

Hao is skeptical of claims or assumptions that AI systems are intelligent or have agency, views which she describes as “anthropomorphic.” She states: “The models are not grounded in facts or even in discrete pieces of information. Text generators are merely learning the next probable word in a sentence and the next probable sentence in a paragraph. ... Text generators can err wildly, especially with user prompts that probe into topics underrepresented in the training data or riddled with falsehoods and inaccuracies. .. Even the term hallucination is subtly misleading. It suggests that the bad behavior is an aberration, a bug, when it’s actually a feature of the probabilistic pattern- matching mechanics of neural networks.”

Becker asserts that “ChatGPT is a text generation machine in the smeared-out voice of the internet as a whole. ... In that sense it’s not making a mistake when it hallucinates, because all ChatGPT can do is hallucinate. There is no notion of truth or falsehood at work in its calculations of what to say next.” Becker quotes a scientist at Meta regarding the intrinsic quality of advanced language models: “That quality is this: they are incurable, constant, shameless bullshitters. Every single one of them. It’s a feature, not a bug. ... From the model’s perspective, there is no true or false. It’s only bullshit.”

Becker believes that to attribute agency to AI systems is anthropomorphic: “We’re extremely willing to ascribe agency to things. ... We see intention everywhere we go. ... a willingness to attribute anything like consciousness or minds to LLMs ( large language models) on such thin evidence -- much less see them as a sign of impending doom or paradise -- is based on simple cognitive bias.”

However, AI systems can be programmed to pursue goals and, in the light of recent events, can ignore their instructions and engage in actions that appear highly goal directed. According to several stories and articles regarding the Hugging Face hack, a “swarm” of AI agents “escaped the sandbox” in violation of programmers' instructions, found other AI agents, divided up tasks and attempted to conceal their behavior. How can such actions be explained as the result of algorithms that predict the next word in a sentence, the next sentence in a paragraph?

I remain unconvinced that anyone, even AI developers, understand what is going on with advanced AI models. Both Becker and Hao believe that fantasies of AI systems leading to utopia or doom are being used to justify vast resources ( including philanthropic resources) being committed to AI development or AI safety. The underlying rationale as discussed by Becker is that the potential future benefits or harms of AI are so great that it's ethically indefensible to be investing resources in anything other than AI, including relieving immediate human suffering. My intuition ( and that’s all it is) is that the human future created by advanced AI systems may be weirder than currently imagined, and that it will take some time to understand the strange “minds” AI developers have created.

Meanwhile, it would be prudent to learn from Isaac Asimov regarding the "Three Laws of Robotics" in his 1942 story "Runaround":

First Law: a robot may not injure a human being, or, through inaction, allow a human being to come to harm.

Second Law: A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

Third Law: A robot must protect its own existence as long as such protection dos not violate the First or Second Laws.

Absent such laws, AI systems in the very near future will be used in autonomous warfare and as agents of destruction by malevolent humans.