Are Early Layers of Encoder-based Large Language Models Effective in Code Classification? 


Vol. 52,  No. 8, pp. 654-659, Aug.  2025
10.5626/JOK.2025.52.8.654


PDF

  Abstract

Encoder-based models are used in code classification due to their ability to effectively represent data. A recently proposed methodology, EarlyBIRD, demonstrated that using the outputs from the early layers of encoder-based models can effectively perform the given task. However, this study only used the CodeBERT model and showed its effectiveness in specific tasks. In this paper, we apply EarlyBIRD to various tasks using the encoder-decoder-based CodeT5 model and discuss its effects. Experimental results showed a 13.79%p performance improvement when the language model was not pre-trained on the programming language of the task, but only a 0.41%p improvement when pre-trained on a similar language. Additionally, the performance of the encoder-decoder-based model without applying EarlyBIRD was similar to the best performance of encoder-based models with EarlyBIRD. It was also found that EarlyBIRD was not effective because it was difficult to pre-select which early layers should be used.


  Statistics
Cumulative Counts from November, 2022
Multiple requests among the same browser session are counted as one view. If you mouse over a chart, the values of data points will be shown.


  Cite this article

[IEEE Style]

C. Lee, S. Ji, H. Im, "Are Early Layers of Encoder-based Large Language Models Effective in Code Classification?," Journal of KIISE, JOK, vol. 52, no. 8, pp. 654-659, 2025. DOI: 10.5626/JOK.2025.52.8.654.


[ACM Style]

Changsup Lee, Suhwan Ji, and Hyeonseung Im. 2025. Are Early Layers of Encoder-based Large Language Models Effective in Code Classification?. Journal of KIISE, JOK, 52, 8, (2025), 654-659. DOI: 10.5626/JOK.2025.52.8.654.


[KCI Style]

이창섭, 지수환, 임현승, "인코더 기반 거대 언어 모델의 초기 계층이 코드 분류에 효과적인가?," 한국정보과학회 논문지, 제52권, 제8호, 654~659쪽, 2025. DOI: 10.5626/JOK.2025.52.8.654.


[Endnote/Zotero/Mendeley (RIS)]  Download


[BibTeX]  Download



Search




Journal of KIISE

  • ISSN : 2383-630X(Print)
  • ISSN : 2383-6296(Electronic)
  • KCI Accredited Journal

Editorial Office

  • Tel. +82-2-588-9240
  • Fax. +82-2-521-1352
  • E-mail. chwoo@kiise.or.kr