ChatGPT runs 10K Nvidia training GPUs

Feb 20, 2023 · chatgpt ·

OpenAI, a research lab dedicated to advancing artificial intelligence (AI), recently announced that its latest AI language model, ChatGPT, is being trained on 10,000 NVIDIA GPUs. The news is particularly significant since it marks a major milestone for OpenAI in the world of AI research and development.

The goal of ChatGPT is to develop an AI-based natural language generation technology that can generate human-like conversations from user input. The training is being conducted using the system's proprietary training software, allowing OpenAI to manipulate more than 8 billion parameters—the equivalent of a network of 100 trillion neurons. Through the use of this system, OpenAI hopes to create a natural language understanding engine that can be used in various applications, including virtual agents, customer service bots, and automated conversational agents.

The training platform consists of 10,000 NVIDIA V100 GPUs, connected by the IBM POWER9 processor and Mellanox InfiniBand networking. This combination enables AI models to access the large datasets required for deep learning training at high speeds, while providing the necessary power efficiency to ensure no performance is lost due to the network connections. Furthermore, this setup also allows the model to be scaled up to potentially thousands more GPUs as OpenAI’s research develops.

OpenAI plans to open source the ChatGPT model after completion of its initial training and validation phases, making it available for other researchers and developers to use and build upon. The release of such a powerful technology is expected to provide a huge boost to the AI industry, as well as facilitate further progress in language understanding and natural conversation generation.

Overall, the introduction of ChatGPT represents a major breakthrough for OpenAI and for the AI industry as a whole. By harnessing the speed and power of 10,000 NVIDIA GPUs, OpenAI is able to train its chatbot model with unprecedented accuracy and scale. Furthermore, the open sourcing of the model will enable others to take advantage of its powerful capabilities and further accelerate AI research and development.