Seungmin Yu  Hayun Lee  Dongkun Shin

10.5626/JOK.2024.51.7.583

NVIDIA GPU Sparse Tensor Core N:M sparsity channel permutation pruning

N:M sparsity pruning is an effective approach for compressing deep neural networks by leveraging NVIDIA’s Sparse Tensor Core technology. Despite its effectiveness, this technique is constrained by hardware limitations, leading to fixed compression ratios and increased access to unnecessary input data, and does not adequately address the imbalanced distribution of essential parameters. This paper proposes Hierarchical N:M (HiNM) sparsity, where vector sparsity is applied prior to N:M sparsity for various-levels of sparsity. We also introduce a novel permutation technique tailored for HiNM sparsity, named 2-axis channel permutation (2CP). The experimental results showed that HiNM sparsity achieves a compression ratio twice that of traditional N:M sparsity while reducing latency by an average of 37%.