HUAWEI CONNECT

Huawei’ s OceanStor M900 Context Memory Storage is aimed at solving a key cause of inefficiency when performing AI inference in hyperscale data centres

While compute for AI was a big focus at September’ s HUAWEI CONNECT event in Shanghai, Huawei did not neglect advanced data storage, arguing that it should be seen as a key foundation in the AI era.

According to Huawei, the rapidly increasing context window of cuttingedge reasoning AI models, triggered in part by the many rounds of dialogue involved in agentic AI, is driving explosive growth in the size of KV Cache – AI models’ working memory when performing inference.

AI inference efficiency The industry believes that the traditional method of relying on GPU memory and system memory to store KV Cache is no longer sustainable in terms of computational efficiency, cost and capacity.

The AI industry’ s typical storage hierarchy for inference( starting with the fastest and most expensive type with the smallest amount of storage) is:

• L1( HBM)

• L2( DRAM)

• L3( local SSD)

• L4( archived / shared storage)

When performing long-sequence inference, KV Cache bottlenecks can occur in high-concurrency scenarios due to insufficient capacity at the L3 layer and inadequate performance at the L4 layer.

While the ideal solution would be to store all KV Cache data in the high-speed L1 and L2 layers, this is too expensive to be practical. Naturally, more people in the industry have come to realise that the solution involves creating a L3.5 layer to fill the gap between expensive GPU memory and slow traditional network storage.

Another issue Huawei highlighted is that with thousands or more compute tasks running simultaneously in ultralarge scale data centres, KV Cache needs to be shared across an entire cluster to eliminate the inefficiencies and waste created by some tasks having memory allocated to them that they aren’ t using or the same data either being loaded into memory or searched for multiple times.

18 huawei. com