Digital Library[ Search Result ]
Korean End-to-End Coreference Resolution with BERT for Long Document
Kyeongbin Jo, Youngjun Jung, Changki Lee, Jihee Ryu, Joonho Lim
http://doi.org/10.5626/JOK.2023.50.1.32
Coreference resolution is a natural language processing task that identifies mentions that are coreference resolution targets, identifies mentions that refer to the same entity, and groups them together. Recently, in coreference resolution, an end-to-end model using BERT to derive the context expression of a word while simultaneously performing mention detection and coreference resolution has been mainly studied. However, BERT has the problem of reduced performance for long documents due to its input length limit. Therefore, in this paper, the following model is proposed. First, a lengthy document is split into tokens of 512 or fewer tokens, extracted from an existing local BERT to obtain the primary contextual expression of a word, and then recombined to compute and add a globalpositional embedding value for the original document. Finally, a coreference resolution was performed by computing the entire context expression with the Global BERT layer. As a result of the experiment, the model proposed in this paper showed similar performance to the existing model, while the GPU memory usage decreased by 1.4 times and the speed improved by 2.1 times.
BERT-based Two-Stage Classification Models and Co-Attention Mechanism for Diagnosing Dementia and Schizophrenia-related Disease
Min-Kyo Jung, Seung-Hoon Na, Ko Woon Kim, Byoung-Soo Shin, Young-Chul Chung
http://doi.org/10.5626/JOK.2022.49.12.1071
Noting the recently increasing number of patients, we present deep learning methods for automatically diagnosing dementia and schizophrenia by exploring the use of the novel two-stage classification and the co-attention mechanism. First, the two-stage classification consists of two steps-the perplexity-based classification and the standard BERT-based classification. 1) the perplexity-based classification first prepares two types of BERTs, i.e., control-specific and patients-specific BERTs, pretrained from transcripts for controls and patients as the additional pretraining datasets, respectively, and then performs a simple threshold-based classification based on the difference between perplexity values of two BERTs for an input test transcript; then, for ambiguous cases where the perplexity difference only does not provide sufficient evidence for the classification, the standard BERT-based classification is performed based on a fine-tuned BERT. Second, the co-attention mechanism enriches the BERT-based representations from a doctor’s transcript and a client’s one by applying the cross-attention over them using the shared affinity matrix, and performs the classification based on the enriched co-attentive representations. Experiment results on a large-scale dataset of Korean transcripts show that the proposed two-stage classification outperforms the baseline BERT model on 4 out of 7 subtasks and the use of the co-attention mechanism achieves the best F1 score for 4 out of 8 subtasks.
Contract Eligibility Verification Enhanced by Keyword and Contextual Embeddings
Sangah Lee, Seokgi Kim, Eunjin Kim, Minji Kang, Hyopil Shin
http://doi.org/10.5626/JOK.2022.49.10.848
Contracts need to be reviewed to be verified if they include all the essential clauses for them to be valid. Such clauses are highly formal and repetitive regardless of the kinds of contracts, and automated legal technologies are required for legal text comprehension. In this paper, we have constructed a simple item-by-item classification model for clauses in contracts to estimate contract eligibility by addressing formal and repetitive properties of contract clauses. We have used keyword embeddings based on conventional requirements of contracts and concatenate them to sentence embeddings of clauses, extracted from a BERT model fine-tuned with legal documents. The contract eligibility can be verified by the predicted labels. Based on our methods, we report reasonable performances with the accuracy of 90.57 and 90.64, and an F1-score of 93.27 and 93.26, using additional keyword embeddings with BERT embeddings.
KorSciQA 2.0: Question Answering Dataset for Machine Reading Comprehension of Korean Papers in Science & Technology Domain
Hyesoo Kong, Hwamook Yoon, Mihwan Hyun, Hyejin Lee, Jaewook Seol
http://doi.org/10.5626/JOK.2022.49.9.686
Recently, the performance of the Machine Reading Comprehension(MRC) system has been increased through various open-ended Question Answering(QA) task, and challenging QA task which has to comprehensively understand multiple text paragraphs and make discrete inferences is being released to train more intelligent MRC systems. However, due to the absence of a QA dataset for complex reasoning to understand academic information in Korean, MRC research on academic papers has been limited. In this paper, we constructed a QA dataset, KorSciQA 2.0, for the full text including abstracts of Korean academic papers and divided the difficulty level into general, easy, and hard for discriminative MRC systems. A methodology, process, and system for constructing KorSciQA 2.0 were proposed. We conducted MRC performance evaluation experiments and when fine-tuning based on the KorSciBERT model, which is a Korean-based BERT model for science and technology domains, the F1 score was 80.76%, showing the highest performance.
Comparison of BERT-based Model Performance in CBCA Criteria Classification
Junho Shin, Jungsoo Shin, Eunkyung Jo, Yeohoon Yoon, Jaehee Jung
http://doi.org/10.5626/JOK.2022.49.9.727
In the case of child sex crimes, the victim"s statement plays a critical role in determining the existence or innocence of the case, so the Supreme Prosecutors" Office classifies the statement into a total of 19 criteria according to Criteria-Based Content Analysis (CBCA), a victim"s statement analysis technique. However, this may differ in criteria classification according to the subjective opinion of the statement analyst. Thus, in this paper, two major classification methods were applied and analyzed to present an criteria classification model using BERT and RoBERTa. The two methods comprise of a method of classifying the entire criterion at the same time, as well as method of dividing it into four groups, and then classifying the criteria within the group secondarily. The experiment classified statements into 16 criteria of CBCA and performed comparative analysis using several pre-trained models. As a result of the classification, the former classification method performed better than the latter classification method in 13 of the total 16 criteria, and the latter method was effective in three criteria with a relatively insufficient number of training data. Additionally, the RoBERTa-based model performed better than the BERT-based model in 15 of the 16 criteria, and the BERT model, which was pre-trained using only Korean conversational colloquial language, classified the remaining one criterion uniquely. This paper shows that the proposed model, which was pre-trained using interactive colloquial data is effective in classifying children"s statement sentences.
KcBert-based Movie Review Corpus Emotion Analysis Using Emotion Vocabulary Dictionary
Yeonji Jang, Jiseon Choi, Hansaem Kim
http://doi.org/10.5626/JOK.2022.49.8.608
Emotion analysis is the classification of human emotions expressed in text data into various emotional types such as joy, sadness, anger, surprise, and fear. In this study, using the emotion vocabulary dictionary, the emotions expressed in the movie review corpus were classified into nine categories: joy, sadness, fear, anger, disgust, surprise, interest, boredom, and pain to construct an emotion corpus. Then, the performance of the model was evaluated by training the emotion corpus in KcBert. To build the emotion analysis corpus, an emotion vocabulary dictionary based on a psychological model was used. It was judged whether the vocabulary of the emotion vocabulary dictionary and the emotion vocabulary displayed in the movie review corpus matched, and the emotion type matching the vocabulary appearing at the end of the movie review corpus was tagged. Based on the performance of the emotion analysis corpus constructed in this way by training it on KcBert pre-trained with NSMC, KcBert showed excellent performance in the model classified into 9 types.
Quality Estimation of Machine Translation using Dual-Encoder Architecture
Dam Heo, Wonkee Lee, Jong-Hyeok Lee
http://doi.org/10.5626/JOK.2022.49.7.521
Quality estimation (QE) is the task of estimating the quality of given machine translations (MTs) without their reference translations. A recent research trend is to apply transfer learning to a pre-training model based on Transformer encoder with a parallel corpus in QE. In this paper, we proposed a dual-encoder architecture that learns a monolingual representation of each respective language in encoders. Thereafter, it learns a cross-lingual representation of each language in cross-attention networks. Thus, it overcomes the limitations of a single-encoder architecture in cross-lingual tasks, such as QE. We proved that the dual-encoder architecture is structurally more advantageous over the single-encoder architecture and furthermore, improved the performance and stability of the dual-encoder model in QE by applying the pre-trained language model to the dual-encoder model. Experiments were conducted on WMT20 QE data for En-De pair. As pre-trained models, our model employs English BERT (Bidirectional Encoder Representations from Transformers) and German BERT to each encoder and achieves the best performance.
SMERT: Single-stream Multimodal BERT for Sentiment Analysis and Emotion Detection
Kyeonghun Kim, Jinuk Park, Jieun Lee, Sanghyun Park
http://doi.org/10.5626/JOK.2021.48.10.1122
Sentiment Analysis is defined as a task that analyzes subjective opinion or propensity and, Emotion Detection is the task that finds emotions such as ‘happy’ or ‘sad’ from text data. Multimodal data refers to the appearance of image and voice data in addition to text data. In prior research, RNN or cross-transformer models were used, however, RNN models have long-term dependency problems. Also, since cross-transformer models could not capture the attribute of modalities, they got worse results. To solve those problems, we propose SMERT based on a single-stream transformer ran on a single network. SMERT can get joint representation for Sentiment Analysis and Emotion Detection. Besides, we use BERT tasks which are improved to utilize for multimodal data. To present the proposed model, we verify the superiority of SMERT through a comparative experiment on the combination of modalities using the CMU-MOSEI dataset and various evaluation metrics.
Combining Sentiment-Combined Model with Pre-Trained BERT Models for Sentiment Analysis
http://doi.org/10.5626/JOK.2021.48.7.815
It is known that BERT can capture various linguistic knowledge from raw text via language modeling without using any additional hand-crafted features. However, some studies have shown that BERT-based models with an additional use of specific language knowledge have higher performance for natural language processing problems associated with that knowledge. Based on such finding, we trained a sentiment-combined model by adding sentiment features to the BERT structure. We constructed sentiment feature embeddings using sentiment polarity and intensity values annotated in a Korean sentiment lexicon and proposed two methods (external fusing and knowledge distillation) to combine sentiment-combined model with a general-purpose BERT pre-trained model. The external fusing method resulted in higher performances in Korean sentiment analysis tasks with movie reviews and hate speech datasets than baselines from other pre-trained models not fused with sentiment-combined models. We also observed that adding sentiment features to the BERT structure improved the model’s language modeling and sentiment analysis performance. Furthermore, when implementing sentiment-combined models, training time and cost could be decreased by using a small-scale BERT model with a small number of layers, dimensions, and steps.
A Leader’s Final Decision Classification Model Tested on Meeting Records with BERT
http://doi.org/10.5626/JOK.2021.48.5.568
The ways in which leaders make decisions affect the performance of the group. To understand these decision-making processes, we first formalize the problem as predicting leaders" decisions from discussion with group members. For this purpose, we introduce conversational meeting records from the annals of the Joseon dynasty. Using this dataset, we develop a Conversational Decision Making Model with BERT (CDMM-B). CDMM-B is a hierarchical structure of RNN and BERT which simultaneously uses both words and speakers. CDMM-B outperforms other baselines in predicting leaders" final decisions. We also investigate the importance of speakers and the order of utterances for the task through an ablation study.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr