Digital Library[ Search Result ]
Lexical Substitution Using a Replaced Token Detection Model
http://doi.org/10.5626/JOK.2023.50.4.321
Substitutes in a sentence are words that do not change the meaning of the sentence if substituted. The task of substitution, also known as lexical substitution, can be applied to various natural language processing tasks, such as data augmentation. Traditional methods for lexical substitution may generate unnatural substitutes. To solve this problem, we propose a new method of lexical substitution. Our method samples sentences containing the target word from a corpus, inputs these sentences to the substitutes generator, which is based on the pretrained BERT, and excludes unacceptable candidates with the replaced token detection model. Verifying the proposed method with the open corpus provided by the National Institute of Korean Language and the Natmal synonym dictionary, our method extracts more accurate substitutes than traditional methods. Also, it is found that the replaced token detection model, which is proposed for lexical substitution, performs better in our experiment than the model learned by using the CoLA dataset, which can be considered to exclude unacceptable candidates.
Data Augmentation Methods for Improving the Performance of Machine Reading Comprehension
Sunkyung Lee, Eunseong Choi, Seonho Jeong, Jongwuk Lee
http://doi.org/10.5626/JOK.2021.48.12.1298
Machine reading comprehension is a method of understanding the meaning and performing inference over a given text by computers, and it is one of the most essential techniques for understanding natural language. The question answering task yields a way to test the reasoning ability of intelligent systems. Nowadays, machine reading comprehension techniques performance has significantly improved following the recent progress of deep neural networks. Nevertheless, there may be challenges in improving performance when data is sparse. To address this issue, we leverage word-level and sentence-level data augmentation techniques through text editing, while minimizing changes to the existing models and cost. In this work, we propose data augmentation methods for a pre-trained language model, which is most widely used in English question answering tasks, to confirm the improved performance over the existing models.
Knowledge Graph Completion using Hyper-class Information and Pre-trained Language Model
http://doi.org/10.5626/JOK.2021.48.11.1228
Link prediction is a task that aims to predict missing links in knowledge graphs. Recently, several link prediction models have been proposed to complete the knowledge graphs and have achieved meaningful results. However, the previous models used only the triples" internal information in the training data, which may lead to an overfitting problem. To address this problem, we propose Hyper-class Information and Pre-trained Language Model (HIP) that performs hyper-class prediction and link prediction through a multi-task learning. HIP learns not only contextual relationship of triples but also abstractive meanings of entities. As a result, it learns general information of the entities and forces the entities connected to the same hyper-class to have similar embeddings. Experimental results show significant improvement in Hits@10 and Mean Rank (MR) compared to KG-BERT and MTL-KGC.
Calibration of Pre-trained Language Model for the Korean Language
Soyeong Jeong, Wonsuk Yang, ChaeHun Park, Jong C. Park
http://doi.org/10.5626/JOK.2021.48.4.434
The development of deep learning models has continuously demonstrated performance beyond humans reach in various tasks such as computer vision and natural language understanding tasks. In particular, pre-trained Transformer models have recently shown remarkable performance in natural language understanding problems such as question answering (QA) tasks and dialogue tasks. However, despite the rapid development of deep learning models such as Transformer-based models, the underlying mechanisms of action remain relatively unknown. As a method of analyzing deep learning models, calibration of models measures the extent of matching of the predicted value of the model (confidence) with the actual value (accuracy). Our study aims at interpreting pre-trained Korean language models based on calibration. In particular, we have analyzed whether pre-trained Korean language models can capture ambiguities in sentences and applied the smoothing methods to quantitatively measure such ambiguities with confidence. In addition, in terms of calibration, we have evaluated the capability of pre-trained Korean language models in identifying grammatical characteristics in the Korean language, which affect semantic changes in the Korean sentences.
Korean Semantic Role Labeling with BERT
Jangseong Bae, Changki Lee, Soojong Lim, Hyunki Kim
http://doi.org/10.5626/JOK.2020.47.11.1021
Semantic role labeling is an application of natural language processing to identify relationships such as "who, what, how and why" with in a sentence. The semantic role labeling study mainly uses machine learning algorithms and the end-to-end method that excludes feature information. Recently, a language model called BERT (Bidirectional Encoder Representations from Transformers) has emerged in the natural language processing field, performing better than the state-of- the-art models in the natural language processing field. The performance of the semantic role labeling study using the end-to-end method is mainly influenced by the structure of the machine learning model or the pre-trained language model. Thus, in this paper, we apply BERT to the Korean semantic role labeling to improve the Korean semantic role labeling performance. As a result, the performance of the Korean semantic role labeling model using BERT is 85.77%, which is better than the existing Korean semantic role labeling model.
A Small-Scale Korean-Specific BERT Language Model
Sangah Lee, Hansol Jang, Yunmee Baik, Suzi Park, Hyopil Shin
http://doi.org/10.5626/JOK.2020.47.7.682
Recent models for the sentence embedding use huge corpus and parameters. They have massive data and large hardware and it incurs extensive time to pre-train. This tendency raises the need for a model with comparable performance while economically using training data. In this study, we proposed a Korean-specific model KR-BERT, using sub-character level to character-level Korean dictionaries and BidirectionalWordPiece Tokenizer. As a result, our KR-BERT model performs comparably and even better than other existing pre-trained models using one-tenth the size of training data from the existing models. It demonstrates that in a morphologically complex and resourceless language, using sub-character level and BidirectionalWordPiece Tokenizer captures language-specific linguistic phenomena that the Multilingual BERT model missed.
Analysis of the Semantic Answer Types to Understand the Limitations of MRQA Models
Doyeon Lim, Haritz Puerto San Roman, Sung-Hyon Myaeng
http://doi.org/10.5626/JOK.2020.47.3.298
Recently, the performance of Machine Reading Question Answering (MRQA) models has surpassed humans on datasets such as SQuAD. For further advances in MRQA techniques, new datasets are being introduced. However, they are rarely based on a deep understanding of the QA capabilities of the existing models tested on the previous datasets. In this study, we analyze the SQuAD dataset quantitatively and qualitatively to demonstrate how the MRQA models answer the questions. It turns out that the current MRQA models rely heavily on the use of wh-words and Lexical Answer Types (LAT) in the questions instead of using the meanings of the entire questions and the evidence documents. Based on this analysis, we present the directions for new datasets so that they can facilitate the advancement of current QA techniques centered around the MRQA models.
A Retrieval Augmented Generation(RAG) System Using Query Rewritting Based on Large Langauge Model(LLM)
Minsu Han, Seokyoung Hong, Myoung-Wan Koo
http://doi.org/10.5626/JOK.2025.52.6.474
This paper proposes a retrieval pipeline that can be effectively utilized in fields requiring expert knowledge without requiring fine-tuning. To achieve high accuracy, we introduce a query rewriting retrieval method that leverages large language models to generate examples similar to the given question, achieving higher similarity than existing retrieval models. The proposed method demonstrates excellent performance in both automated evaluations and expert qualitative assessments, while also providing explainability in retrieval results through generated examples. Additionally, we suggest prompts that can be utilized in various domains requiring specialized knowledge during the application of this method. Furthermore, we propose a pipeline method that incorporates a Top-1 retrieval model, which chooses the most relevant document from the three returned by the query rewriting retrieval model. This aims to prevent the hallucination issue caused by the input of unnecessary documents into the large language model.
Hierarchical Semantic Prompt Design for Robust Open-Vocabulary Object Detection
http://doi.org/10.5626/JOK.2025.52.6.499
Open-Vocabulary Object Detection (OVOD) has been proposed to overcome the limitation of traditional object detection methods, which are restricted to recognizing only categories seen during training. While conventional OVOD approaches generate classifiers using simple prompts like “a {category}”, this paper incorporates the hierarchical structure of object categories into prompts to enhances detection performance. Specifically, we applied prompt engineering techniques that could reduce the use of lengthy connectives and place important keywords at the beginning of the sentence. This resulted in more effective prompts that could capture the intrinsic meaning of hierarchical information. Our method allows for the generation of classifiers without additional computational resources or retraining. Furthermore, it demonstrates strong generalizability. It can be applied to other tasks such as image captioning and medical image analysis. By leveraging hierarchical expressions familiar to humans, our approach also contributes to improving the interpretability of model outputs.
A Large Language Model-based Multi-domain Recommender System using Model Merging
http://doi.org/10.5626/JOK.2025.52.6.548
Recent research in recommender systems has increasingly focused on leveraging pre-trained large language models (LLMs) to effectively understand the natural language information associated with recommendation items. While these LLM-based recommender systems achieve high accuracy, they have a limitation in that they require training separate recommendation models for each domain. This increases the costs of storing and inferring multiple models and makes it difficult to share knowledge across domains. To address this issue, we propose an LLM-based recommendation model that effectively operates across diverse recommendation domains by applying task vector-based model merging. During the merging process, knowledge distillation is utilized from individually trained domain-specific recommendation models to learn optimal merging weights. Experimental results show that our proposed method improves recommendation accuracy by an average of 2.75% across eight domains compared to recommender models utilizing existing model merging methods, while also demonstrating strong generalization performance in previously unseen domains.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr