Calibration of Pre-trained Language Model for the Korean Language

Soyeong Jeong, Wonsuk Yang, ChaeHun Park, Jong C. Park

http://doi.org/10.5626/JOK.2021.48.4.434

The development of deep learning models has continuously demonstrated performance beyond humans reach in various tasks such as computer vision and natural language understanding tasks. In particular, pre-trained Transformer models have recently shown remarkable performance in natural language understanding problems such as question answering (QA) tasks and dialogue tasks. However, despite the rapid development of deep learning models such as Transformer-based models, the underlying mechanisms of action remain relatively unknown. As a method of analyzing deep learning models, calibration of models measures the extent of matching of the predicted value of the model (confidence) with the actual value (accuracy). Our study aims at interpreting pre-trained Korean language models based on calibration. In particular, we have analyzed whether pre-trained Korean language models can capture ambiguities in sentences and applied the smoothing methods to quantitatively measure such ambiguities with confidence. In addition, in terms of calibration, we have evaluated the capability of pre-trained Korean language models in identifying grammatical characteristics in the Korean language, which affect semantic changes in the Korean sentences.