Efficiently Lightweight Korean Language Model with Post-layer Pruning and Multi-stage Fine-tuning

Jae Seong Kim, Suan Lee

http://doi.org/10.5626/JOK.2025.52.3.260

The increasing size of large-scale language models has led to the need for lightweighting for practical applications. This study presents a method to reduce the existing 8B model to 5B by late-layer pruning, while maintaining and improving its performance through two phases of fine-tuning. In the broad fine-tuning phase, we expanded the model's ability to understand and generate Korean by utilizing English-Korean parallel data and a large Korean corpus, and in the refined fine-tuning phase, we enhanced its expressive and inferential capabilities with high-quality datasets. In addition, we integrated the strengths of individual models through model merging techniques. In the LogicKor leaderboard evaluation, the proposed model performed well in the areas of reasoning, writing, and comprehension, with an overall score of 4.36, outperforming the original Llama-3.1-8B-Instruct model (4.35). This demonstrates a 37.5% reduction in model size while still improving performance.