Semiconductor technologies evolving to meet generative AI demand

Power and processor-hungry AI set to drive next stage of chip developmentGenerative artificial intelligence (AI) is redefining the value of work, and its exponential need for performance is driving demand for power consumption and processing capacity in ways that it is not yet clear how this will be met.

Tools based on generative AI have moved beyond niche technical applications to consumer use, changing the way images, video, music, and other content are produced. The impact is expected to reach almost every professional role, including lawyers, architects, doctors and more.

The AI server market is expected to expand at a compound annual growth rate of 36% from 2023 to 2029, outpacing overall data center growth.

Jérôme Azémar, Sales Force Effectiveness Director at Yole Group, recently attended the International Semiconductor Executive Summit (ISES) Taiwan and among the key themes that emerged are the evolution in semiconductor technologies to meet demand.

Increasing processor capacity driving power demand

The massive growth that the data center GPU and AI ASIC market experienced in 2023 — climbing by 167% year on year, is expected to continue in 2024, before stabilizing in 2025.

Yole Group expects this stabilization as the number of companies able to buy GPUs and AI ASICs on a large scale is limited, and as the lifecycle of these components is also growing on average. The total market is expected to reach more than $230 billion by 2029 from more than $150 billion in 2025, representing a CAGR of 29%.

The explosion in the total number of FLOPs used to train AI models over the past decade has driven up the cost into the hundreds of millions of dollars and power consumption from a few kilowatts into megawatts.

Companies are now focusing on increasing power efficiency, as there is currently not enough power generation on the grid to support the rapid pace of growth.

How will HBM4 evolve?   

Generative AI and high-performance computing (HPC) are fueling growth in data center dynamic random-access memory (DRAM) demand.

Yole Group predicts that high-bandwidth memory (HBM) bit shipments will rise by around 48% between 2023 and 2029, with HBM’s share of the DRAM market rising above 6% from around 2% in 2023.

A Yole Group’s teardown shows that NVIDIA has introduced an innovative design in its Hopper Tensor Core H100 GPU that uses HBM3 offering twice as much DRAM bandwidth as its predecessor, the A100. The GPU is paired with the Grace CPU using NVIDIA’s ultra-fast chip-to-chip interconnect, delivering 900GB/s of bandwidth, to provide up to 10 times higher performance for applications running terabytes of data for the HPC, AI, and gaming markets.

HBM is in the hands of three main players – SK Hynix, Samsung and Micron. SK Hynix is likely to be the first to introduce HBM4 in 2026, but it remains to be seen if hybrid bonding will be used . Next-generation hybrid bonding enables the use of smaller connections for more dense stacks.

Jérôme Azémar Sales Force Effectiveness Director at Yole Group
The need for HBM4 is clear, but it is still unclear whether hybrid bonding will be used to achieve the manufacturing. It will be dependent on entry barrier cost and if yields are high enough..

Packaging trends towards glass substrates and chiplets

The current average substrate size of AI accelerators is around 70*70 mm² and the next-generation products are expected to scale to larger sizes, but this trend starts to be limited at 100*100 mm². Beyond this limit, the yield drops drastically.

Not all the current AI accelerators are using the maximum layer count, but next-generation products are expected to quickly adopt higher layer count to face the signal routing and power delivery challenges in parallel with an increased form factor.

One approach to increasing scalability could take the advanced substrates to the next level by adopting glass as a new core material. Glass core substrates can offer smaller L/S, fewer layers, and superior thermal conductivity, making them a promising candidate to resolve thermal management issues.

Yole Group expects to see the chiplet approach take hold in the future as manufacturers look to optimize cost against performance. Chiplets enable disaggregation so that the system-on-chip (SoC) monolithic die is partitioned into smaller chips with different functions and then interconnected in the same package. Chiplets also allow for duplication so that two or more SoC monolithic dies can interconnected in the same package to form a larger SoC.

Jérôme Azémar from Yole Group
This allows designers to place most advanced functions requiring the smallest nodes in small dedicated chips, rather than having a large chip requiring the smallest node to handle all functions. If you can have some of the functions handled by other chips with less advanced nodes, then total cost would be lower.. So, it is becoming a plug-and-play approach where you need to have the optimum case for each and it’s a matter of balance between the space you want to take up, the cost of the chip, the performance you want to obtain and how you pack it.

Taiwan’s central role in generative AI

While Taiwan does not have large chipmaking IDMs in its territory, the country plays a strong role in the AI ecosystem. It serves the likes of AMD and NVIDIA and has key players in processor foundries and packaging integration.

With the presence of TSMC, Taiwan is home to the main contributing foundry for generative AI. Taiwan is also essential in packaging, with TSMC being global leader and other players such as ASE and PTI well positioned for other chip packaging for data centers. Leading players in substrates involved in AI such as Unimicron) are also present in Taiwan.

The HBM market is dominated by Samsung, SK Hynix and Micron, but Taiwan has other important players including Nanya and Winbond, which are involved in supplying other types of memory for data centers. At the server level, Taiwan has strong companies too such as Quanta Computer.

Keep following as Yole Group continues to track the impact of generative AI on semiconductor technologies.

Related events

The neuromorphic sensing and computing markets are projected to experience substantial growth in the coming years, reaching a combined value of $8.4B by 2034. Mobile applications will drive the neuromorphic sensing market, while data center applications will lead in the neuromorphic computing market. The neuromorphic ecosystem is maturing, with key players and startups adopting diverse strategies to capitalize on market opportunities…. Program & & registration

About the author

Jérôme Azémar is Sales Force Effectiveness Director at Yole Group.

Jérôme’s mission is to develop Yole Group’s business in the semiconductor industry, enhance sales activities, create and maintain long-term relationships with Yole Group’s accounts, and meet their expectations. He coordinates and optimizes sales operations at Yole Group in line with the company’s business interests while focusing on our customers’ satisfaction. He is involved in all the company’s coverage within the semiconductor industry, from manufacturing to packaging.

Jérôme is the author of numerous analyses and international publications covering advanced packaging, power electronics, and semiconductor manufacturing. He is also regularly involved in international conferences, giving presentations, delivering keynotes, as well as organizing committees.

Prior to this and upon graduating from INSA Toulouse (France) with a Master’s in Microelectronics and Applied Physics, Jérôme joined ASML and worked in Veldhoven (The Netherlands) for three years as an Application Support Engineer specializing in immersion scanners. During this time, he acquired photolithographic skills which he then honed during a two-year stint as a process engineer at STMicroelectronics (France).

This article has been developed in collaboration with the More Moore and Semiconductor Packaging teams at Yole Group, including semiconductor packaging, memory and computing activities.

Related products