Artificial intelligence (AI) developers may soon face a shortage of text to train their generative AI models, warned Stuart Russell, a professor at the University of California, Berkeley. Speaking at a UN summit, Russell highlighted that AI's reliance on extensive text data for training large language models is starting to encounter limitations.
Russell expressed concern that AI-powered bots like ChatGPT could be "running out of text in the universe" that serves as their training material. The technology that fuels these bots, which analyzes vast amounts of digital text, is now reaching a critical point. During an interview with the International Telecommunication Union, a UN communications agency, Russell described the situation as AI's progress hitting a "brick wall."
The scarcity of available text for training could potentially impact generative AI developers in their data collection and training approaches in the near future. These challenges arise as AI continues to advance in replacing human involvement in tasks primarily focused on language.
Despite the impending text shortage, Russell maintains his belief that AI will increasingly replace humans in numerous jobs that involve a "language in, language out" dynamic. While the limitations on available text may pose hurdles, he remains optimistic about the potential of AI to assume language-related tasks traditionally performed by humans.
How are large language models trained?
In an exclusive interview with Business Today, Stuart Russell explained how large language models like GPT-4 are trained using massive amounts of text.
Russell explained, "So basically, lots of sequences of words are collected from various digital resources. In the case of GPT-4, we think somewhere in the 20 to 30 trillion words of text, which is approximately the same amount as we have in all the books that the human race has ever written. And then we start from what you might think of as a blank slate, an enormous circuit with about a trillion parameters or more. And then by the process of doing about a billion trillion, small random permutations to those parameters, the system is gradually improved, and its ability to predict the next word. So the result of that is something that when you converse with it has many the appearances of a really intelligent entity."
Dip in ChatGPT Traffic
The overall training of chatbots may hit a brick wall in the future but in terms of usage, ChatGPT has witnessed a drop in active users, since the peak it reached after the launch.
According to analytics firm Similarweb, global desktop, and mobile traffic to the ChatGPT website declined by 9.7 per cent in June compared to May. Unique visitors also dropped by 5.7 per cent, and the average time spent on the website decreased by 8.5 per cent. Similarweb's Senior Insights Manager, David Carr, speculated that this decline could signify a decrease in interest in the chatbot. However, Rishi Jaluria, an analyst at RBC Capital Markets, interpreted the data differently, suggesting that it reflects a growing demand for generative AI that offers real-time information.