Kim Ho-sik, SK hynix executive vice president
Speaks at Semicon Taiwan 2026
'AI bottleneck is memory, not GPU'
HBM expanding from storage to computing
![Kim Ho-sik, executive vice president of memory systems research at SK hynix, delivers a presentation at FMS (the Future Memory and Storage Conference) in 2024. [Captured from LinkedIn]](https://wimg.heraldcorp.com/news/cms/2026/09/02/news-p.v1.20260902.fe7ba6876b0e4ad390923c31600204ed_P1.png)
The bottleneck in the AI race is shifting rapidly from GPU computing power to memory capacity and bandwidth. SK hynix said that as AI workloads move from training to inference and agentic AI spreads, the role of memory is expanding beyond simple storage into the realm of computation.
Kim Ho-sik, executive vice president of memory systems research at SK hynix, delivered a presentation titled "How AI Redefines the Role of Memory" at the Semicon Taiwan 2026 Memory Executive Summit in Taiwan on Tuesday, according to industry sources and Taiwanese media.
Kim said that while memory chips were once viewed primarily as components to be optimized for cost and power consumption during the PC and cloud eras, they are now evolving into "strategic infrastructure" that determines system performance, scalability and competitiveness in the AI age.
He added that the value of memory should no longer be measured by raw capacity or speed alone, but evaluated through the lens of overall AI system efficiency — metrics such as tokens per dollar and tokens per watt.
The shift in AI's center of gravity from training to inference is amplifying memory's importance further. Kim cited Nvidia's rack-scale AI system NVL72 as an example, noting that KV cache alone accounts for more than half of the system's total memory footprint.
KV cache is a temporary memory space that stores intermediate data so an AI model does not have to recompute previously processed context. As conversations grow longer and complex reasoning tasks multiply, the memory capacity required for KV cache grows sharply.
Kim also pointed to Nvidia's H200 as a case in point. Without significantly redesigning its compute cores, the H200 increased memory bandwidth by 43 percent and capacity by 76 percent, achieving roughly a 90 percent improvement in inference performance. The example underscores that how much data can be supplied to a GPU — and how quickly — has become as critical a variable as raw compute power in boosting AI performance.
Building on that, Kim said memory's role is expanding beyond "storage" into "computing." He noted that market demand is growing for data computing, memory computing and in-memory computing architectures.
HBM marks the starting point of this transformation. Placing GPUs and HBM chips in close proximity within a single package dramatically reduces the physical distance between the processor and memory.
Combining NPUs with three-dimensionally stacked DRAM follows the same logic. Kim said system architectures are already shifting toward designs that shrink the distance between processors and memory, cutting the time and power needed to move data.
Research is also underway to push HBM's role further — from storage toward active computation. Kim said that starting with HBM4, the use of logic process technology rather than conventional memory process technology in the base die opens up significantly more room to add diverse functions. In custom HBM designs, some computing functions can be embedded in the base die, allowing memory to handle tasks previously performed by the GPU.
StreamDQ, a technology under development at SK hynix, is a prime example. A dedicated compute block embedded in the HBM base die handles the dequantization work that GPUs would otherwise perform during AI inference, reducing both the GPU's workload and the volume of data movement. SK hynix researchers have presented findings showing this architecture can increase token throughput in large language models by up to 5.15 times.
Another pillar of memory innovation is sharing — allowing multiple processors to access the same pool of memory. As AI systems evolve from single-server setups to architectures that link multiple servers and accelerators into one vast system, the inefficiency of each server storing and repeatedly copying the same data is becoming a growing problem.
Kim proposed storing KV cache in a single shared memory pool accessible to multiple CPUs and GPUs simultaneously, rather than having each compute node maintain its own copy. He noted that several large data center operators, including China's Alibaba, have already applied shared memory designs to their production infrastructure. Compute Express Link, or CXL, is the key technology enabling CPUs and GPUs to jointly access external memory resources.
Kim also stressed the growing importance of advanced packaging. In AI systems, the energy required to move data to a processor can be thousands to tens of thousands of times greater than the energy needed for the computation itself, making it a central challenge to reduce both the distance and frequency of data movement.
He presented 2.5D and 3D packaging technologies as the solution. Advancing from conventional 2.5D packaging to 3D packaging — which stacks chips vertically — can further shorten the distance data must travel. Kim said this approach could bring the energy cost of data transmission down to as low as 0.2 to 0.3 picojoules per bit.
Ultimately, Kim said, the role of memory makers themselves must change. Simply producing faster chips with greater capacity, as in the past, will not be enough to remain competitive in the AI era.
He said memory companies must evolve into "co-design partners" that engage with customers from the earliest stages of AI system design, jointly optimizing performance, power consumption and cost.
"We need to maximize customers' cost efficiency and power efficiency at the system architecture level, and support them in meeting demand for rack-scale AI systems and intelligent computing going forward," Kim said.
go@heraldcorp.com
