Digital Library[ Search Result ]
Multi-task Learning Based Re-ranker for External Knowledge Retrieval in Document-grounded Dialogue Systems
http://doi.org/10.5626/JOK.2023.50.7.606
Document-grounded dialogue systems retrieve external passages related to the dialogue and use them to generate an appropriate response to the user"s utterance. However, the retriever based on the dual-encoder architecture records low performance in finding relevant passages, and the re-ranker to complement the retriever is not sufficiently optimized. In this paper, to solve these problems and perform effective external passage retrieval, we propose a re-ranker based on multi-task learning. The proposed model is a cross-encoder structure that simultaneously learns contrastive learning-based ranking, Masked Language Model (MLM), and Posterior Differential Regularization (PDR) in the fine-tuning stage, enhancing language understanding ability and robustness of the model through auxiliary tasks of MLM and PDR. Evaluation results on the Multidoc2dial dataset show that the proposed model outperforms the baseline model in Recall@1, Recall@5, and Recall@10.
Type-specific Multi-Head Shared-Encoder Model for Commonsense Machine Reading Comprehension
http://doi.org/10.5626/JOK.2023.50.5.376
Machine reading comprehension (MRC) is a task introduced to a machine that can understand natural languages by solving various tasks based on given context. To evaluate natural language understanding of machine, a machine must make commonsense inference under full comprehension of a given context. To enhance model obtaining such abilities, we proposed a multi-task learning scheme and a model for commonsense MRC. Contributions of this study are as follows: 1) a method of task-specific dataset configuration is proposed; 2) a type-specific multi-head shared-encoder model with multi-task learning scheme including batch sampling and loss scaling is developed; and 3) when the method is evaluated on CosmosQA dataset (commonsense MRC), the accuracy was improved by 2.38% compared to the performance at baseline with fine-tuning.
Multi-task Learning Approach Based on Pre-trained Language Models Using Temporal Relations
Chae-Gyun Lim, Kyo-Joong Oh, Ho-Jin Choi
http://doi.org/10.5626/JOK.2023.50.1.25
In the research on natural language understanding that can perform multiple tasks and produce a model that provides general performance, various studies of multi-task learning techniques are being attempted. In addition, documents written in natural language typically contain time-related information, and accurate recognize such information is essential to understand the overall content and context of the document. In this paper, we propose a multi-task learning technique that incorporates a temporal relation extraction task into the learning process of NLU tasks to use the temporal contextual information of Korean input sentences. In order to reflect the characteristics of multi-task learning, a new task for extracting temporal relations is designed, and the model is configured to learn in conjunction with existing NLU tasks. In the experiment, the difference in performance was analyzed by learning the effect of various task combinations and the temporal relationships compared to the case where only the existing NLU task is used. Through the experimental results, we discuss that the overall performance of the multi-task combination is higher than that of individual tasks, especially when temporal relationship with the name entity recognition shows greatly improved performance.
Structuralized External Knowledge and Multi-task Learning for Knowledge Selection
http://doi.org/10.5626/JOK.2022.49.10.884
Typically, task-oriented dialog systems use well-structured knowledge, such as databases, to generate the most appropriate responses to users" questions. However, to generate more appropriate and fluent responses, external knowledge, which is unstructured text data such as web data or FAQs, is necessary. In this paper, we propose a novel multi-task learning method with a pre-trained language model and a graph neural network. The proposed method makes the system select the external knowledge effectively by not only understanding linguistic information but also grasping the structural information latent in external knowledge which is converted into structured data, graphs, using a dependency parser. Experimental results show that our proposed method obtains higher performance than the traditional bi-encoder or cross-encoder methods that use pre-trained language models.
Knowledge Graph Completion using Hyper-class Information and Pre-trained Language Model
http://doi.org/10.5626/JOK.2021.48.11.1228
Link prediction is a task that aims to predict missing links in knowledge graphs. Recently, several link prediction models have been proposed to complete the knowledge graphs and have achieved meaningful results. However, the previous models used only the triples" internal information in the training data, which may lead to an overfitting problem. To address this problem, we propose Hyper-class Information and Pre-trained Language Model (HIP) that performs hyper-class prediction and link prediction through a multi-task learning. HIP learns not only contextual relationship of triples but also abstractive meanings of entities. As a result, it learns general information of the entities and forces the entities connected to the same hyper-class to have similar embeddings. Experimental results show significant improvement in Hits@10 and Mean Rank (MR) compared to KG-BERT and MTL-KGC.
Grammatical Error Detection for L2 Learners Based on Attention Mechanism
Chanhee Park, Jinuk Park, Minsoo Cho, Sanghyun Park
http://doi.org/10.5626/JOK.2019.46.6.554
Grammar Error Detection refers to the work of discovering the presence and location of grammatical errors in a given sentence, and is considered to be useful for L2 learners to learn and evaluate the language. Systems for grammatical error correction have been actively studied, but there still exist limitations such as lack of training corpus and limited error type correction. Therefore, this paper proposes a model for generalized grammatical error detection through the sequence labeling problem which does not require the determination of error type. The proposed model dynamically decides character-level and word-level representation to deal with unexpected words in L2 learners" writing. Also, based on the proposed model the bias which can occur during the learning process with imbalanced data can be avoided through multi-task learning. Additionally, attention mechanism is applied to efficiently predict errors by concentrating on words for judging errors. To validate the proposed model, three test data were used and the effectiveness of the model was verified through the ablation experiment.
Korean Dependency Parsing using Pointer Networks
http://doi.org/10.5626/JOK.2017.44.8.822
In this paper, we propose a Korean dependency parsing model using multi-task learning based pointer networks. Multi-task learning is a method that can be used to improve the performance by learning two or more problems at the same time. In this paper, we perform dependency parsing by using pointer networks based on this method and simultaneously obtaining the dependency relation and dependency label information of the words. We define five input criteria to perform pointer networks based on multi-task learning of morpheme in dependency parsing of a word. We apply a fine-tuning method to further improve the performance of the dependency parsing proposed in this paper. The results of our experiment show that the proposed model has better UAS 91.79% and LAS 89.48% than conventional Korean dependency parsing.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr