Are Early Layers of Encoder-based Large Language Models Effective in Code Classification?

Changsup Lee, Suhwan Ji, Hyeonseung Im

http://doi.org/10.5626/JOK.2025.52.8.654

Encoder-based models are used in code classification due to their ability to effectively represent data. A recently proposed methodology, EarlyBIRD, demonstrated that using the outputs from the early layers of encoder-based models can effectively perform the given task. However, this study only used the CodeBERT model and showed its effectiveness in specific tasks. In this paper, we apply EarlyBIRD to various tasks using the encoder-decoder-based CodeT5 model and discuss its effects. Experimental results showed a 13.79%p performance improvement when the language model was not pre-trained on the programming language of the task, but only a 0.41%p improvement when pre-trained on a similar language. Additionally, the performance of the encoder-decoder-based model without applying EarlyBIRD was similar to the best performance of encoder-based models with EarlyBIRD. It was also found that EarlyBIRD was not effective because it was difficult to pre-select which early layers should be used.