The common narrative when it comes to training large language models is the shortage of GPUs. Nvidia’s powerful chips are highly sought after by various AI organizations. However, Elon Musk, the billionaire and tech visionary, presents a different perspective. He believes that the real challenge may not be the lack of GPUs, but rather insufficient power. Musk predicts that the upcoming Grok 3 generation of the AI model from his startup xAI will require approximately 100,000 of Nvidia’s H100 GPUs for training. While acquiring 100,000 H100 GPUs would be a daunting and costly task, the real issue lies in the substantial power consumption of these chips.
Each Nvidia H100 GPU consumes a peak of 700W of power. Therefore, for 100,000 GPUs, the total peak power consumption would be 70 megawatts. Although all 100,000 GPUs may not be running at maximum load simultaneously, the overall power requirements for an AI setup involve more than just the GPUs themselves. Supporting hardware and infrastructure add to the power demands. With 100,000 H100 GPUs, the power consumption could exceed 100 megawatts, equivalent to that of a small city. To put it into perspective, in 2022, the entirety of Paris had 500 megawatts worth of data centers. Therefore, allocating 100 megawatts for just one Large Language Model (LLM) poses a significant challenge.
In an interview with Norway wealth fund CEO Nicolai Tangen on X Spaces, Musk emphasized that while GPU availability remains a major constraint for AI model development, the availability of sufficient electricity will increasingly become a limiting factor. Additionally, Musk made a bold prediction that Artificial General Intelligence (AGI) will surpass human intelligence within the next two years. Despite his confidence in technological advancements, Musk’s track record of predictions has not always been accurate. For example, his forecast in 2017 that self-driving cars capable of allowing passengers to “go to sleep” would be available within two years has not yet materialized. Similarly, his projection in March 2020 that the US would have “close to zero new cases” of Covid-19 by the end of April turned out to be incorrect.
Musk’s predictions may be met with skepticism given his past inaccuracies. Nevertheless, his insight into the GPU requirements for his next-generation LLM is worth considering. The substantial power budget needed for training the Grok 3 model indicates a genuine concern. Notably, xAI’s current model, Grok 2, reportedly only needed 20,000 H100 GPUs, suggesting a five-fold increase in GPU requirements from one model to the next. Such exponential scaling in GPU count and power consumption raises questions about sustainability.
While the GPU shortage is a prevalent issue in training large language models, the escalating power demands pose an equally significant challenge. Elon Musk’s predictions regarding the future of AI and the power requirements for training advanced models shed light on the complexities and limitations of current technology. As the pursuit of Artificial General Intelligence evolves, striking a balance between performance, efficiency, and sustainability will be crucial in shaping the future of AI development.
Leave a Reply