Yole Group Viewpoint: Transforming Data Centers with Generative AI: Spotlight on Memory and Processing Power

In today’s data-driven world, the significance of data centers has escalated to become a critical concern for key entities in the worldwide digital economy. This is especially true as GAFAM (Google, Amazon, Facebook (Meta), Apple, and Microsoft) lead a transformative charge setting in motion a large semiconductor ecosystem. A notable development reshaping the semiconductor industry is the advent of generative AI technology. It is vital to understand thoroughly its impact across the entire supply chain.

As generative AI technologies gain momentum, hyperscale data center operators and original equipment manufacturers (OEMs) are ramping up their server capacities. This expansion is essential for accommodating the computational demands of AI model training and inference processes, necessitating a substantial increase in AI accelerators. The resulting effect is a significant surge in demand for processor units and high bandwidth memory (HBM) associated with these accelerators. In 2023, the requirement for data center accelerators exceeded four million units, with projections indicating a potential doubling of this figure by 2024.

This surge in generative AI applications has also spurred the demand for advanced memory technologies within data centers, such as high-speed DDR5 dynamic random-access memory (DRAM) and HBM memory. These technologies are critical for managing AI workloads, which require enhanced bandwidth capabilities to facilitate faster data exchange between devices and processing units.

Amidst these developments, Nvidia has emerged as a standout performer, establishing itself as a key player while discreetly laying the groundwork in generative AI for several years. A pivotal moment in Nvidia’s strategic expansion into AI and GPU infrastructure enhancement came with its acquisition of Mellanox Technologies in 2019, a deal valued at $7 billion. At the time, Nvidia was generating around $11 billion in revenues annually, indicating the significance of this acquisition in its overall strategy. Mellanox, known for its high-performance computing and networking technology, was integrated into Nvidia’s operations to bolster its GPU and AI infrastructure offerings. This acquisition allowed Nvidia to enhance its data center capabilities significantly, making it a formidable force in AI, particularly in areas requiring intense computational power and efficient data processing.

Nvidia’s operations and ambitions in chip manufacturing are deeply intertwined with TSMC (Taiwan Semiconductor Manufacturing Company), a leading foundry specializing in semiconductor manufacturing. Nvidia relies on TSMC’s advanced technology, notably its CoWoS (Chip on Wafer on Substrate) packaging technology, to produce its high-performance processors. CoWoS is an advanced packaging solution that enables the integration of different types of chips, such as CPU, GPU, and memory, into a single module, significantly enhancing performance and energy efficiency.

However, the soaring demand for Nvidia’s processors, fueled by the growth in AI applications, has led to a bottleneck at TSMC. Despite TSMC’s status as a leading foundry with cutting-edge manufacturing capabilities, it struggles to meet the overwhelming demand from Nvidia and other tech giants, resulting in extended delivery times for chip orders. This situation has opened the door for other major foundry players, such as Samsung, to capitalize on the opportunity. Samsung, with its own advanced semiconductor manufacturing capabilities, is keenly eyeing this space, aiming to attract companies that are facing delays with TSMC and are in search of alternative manufacturing partners to meet their production needs. This dynamic illustrates the competitive and interconnected landscape of the semiconductor industry, where supply chain relationships and manufacturing capacities play critical roles.

Further down the supply chain, companies specializing in advanced packaging seems to experience a limited boost from these technological shifts. This advantage stems from the increased demand for sophisticated packaging solutions that can accommodate the complex requirements of next-generation semiconductor devices.

The impact at the substrate level of the supply chain is still to be measured. Despite the heightened activity in the semiconductor space, substrate manufacturers seem not to have seen a corresponding increase in demand. This situation might be attributed to high substrate inventory levels, keeping their prices low. With the semiconductor shortage coming to an end, the era of skyrocketing substrate prices has also concluded.

The surge in generative AI, especially through Large Language Models (LLMs), has sparked debate over their substantial energy consumption. The shift towards smaller and optimized LLMs for diverse end systems raises hopes for efficiency gains. This trend could accelerate the move towards neuromorphic computing, promising a more energy-efficient approach by emulating the brain’s neural architecture.