Digital Library[ Search Result ]
Post-training Methods for Improving Korean Document Summarization Model
So-Eon Kim, Seong-Eun Hong, Gyu-Min Park, Choong Seon Hong, Seong-Bae Park
http://doi.org/10.5626/JOK.2023.50.10.882
The document summarization task generates a short summary based on a long document. Recently, a method using a pre-trained model based on a transformer model showed high performance. However, as it was proved that fine-tuning does not train the model optimally due to the learning gap between pre-training and fine-tuning, post-training, which is additional training between pre-training and fine-tuning, was proposed. This paper proposed two post-training methods for Korean document summarization. One was Korean Spacing, which is for learning Korean structure, and the other was First Sentence Masking, which is for learning about document summarization. Experiments proved that the proposed post-training methods were effective as performance improved when the proposed post-training was used compared to when it was not.
Multi-Document Summarization Use Semantic Similarity and Information Quantity of Sentence
Yeon-Soo Lim, Sunggoo Kwon, Bong-Min Kim, Seong-Bae Park
http://doi.org/10.5626/JOK.2023.50.7.561
Document summarization task has recently emerged as an important task in natural language processing because of the need for delivering concise information. However, it is difficult to obtain a suitable multi-document summarization dataset. In this paper, rather than training with a multi-document summarization dataset, we propose to use a single-document summarization dataset. That is, we propose a multi-document summarization model which generates multiple single-document summaries with a single-document summarization model and then post-processes these summaries. The proposed model consists of three modules: a summary module, a similarity module, and an information module. When multiple documents are entered into the proposed model, the summary module generates summaries of every single document. The similarity module clusters similar summaries by measuring semantic similarity. The information module selects the most informative summary from each similar summary group and collects selected summaries for the final multi-document summary. Experimental results show that the proposed model outperforms the baseline models and it can generate a high-quality multi-document summary. In addition, the performances of each module also show meaningful results.
Effective Transfer Learning in Text Classification with the Label-Based Discriminative Feature Learning
http://doi.org/10.5626/JOK.2022.49.3.214
The performance of the natural language processing with transfer learning methodology has improved by pre-training language models with a large amount of general data and applying them on downstream tasks. However, the problem is that it learns general features rather than those specific to the downstream tasks as the data used in pre-training is irrelevant to the downstream tasks. This paper proposes a novel learning method for embeddings of pre-trained models to learn specific features of the downstream tasks. The proposed method is to learn the label feature of the downstream tasks through contrast learning with label embedding and sampled data pairs. To demonstrate the performance of the proposed method, we conducted experiments on sentence classification datasets and evaluated whether features of downstream tasks have been learned through PCA(Principal component analysis) and clustering on embeddings.
Korean Text Summarization using MASS with Copying and Coverage Mechanism and Length Embedding
Youngjun Jung, Changki Lee, Wooyoung Go, Hanjun Yoon
http://doi.org/10.5626/JOK.2022.49.1.25
Text summarization is a technology that generates a summary including important and essential information from a given document, and an end-to-end abstractive summarization model using a sequence-to-sequence model is mainly studied. Recently, a transfer learning method that performs fine-tuning using a pre-training model based on large-scale monolingual data has been actively studied in the field of natural language processing. In this paper, we applied the copying mechanism method to the MASS model, conducted pre-training for Korean language generation, and then applied it to Korean text summarization. In addition, coverage mechanism and length embedding were additionally applied to improve the summarization model. As a result of the experiment, it was shown that the Korean text summarization model, which applied the copying and coverage mechanism method to the MASS model, showed a higher performance than the existing models, and that the length of the summary could be adjusted through length embedding.
Topic Centric Korean Text Summarization using Attribute Model
Su-Hwan Yoon, A-Yeong Kim, Seong-Bae Park
http://doi.org/10.5626/JOK.2021.48.6.688
Abstractive summarization takes original text as an input and generates a summary containing the core-information about the original text. The abstractive summarization model is mainly designed by the Sequence-to-Sequence model. To improve quality as well as coherence of summary, the topic-centric methods which contain the core information of the original text are recently proposed. However, the previous methods perform additional training steps which make it difficult to take advantage of the pre-trained language model. This paper proposes a topic-centric summarizer that can reflect topic words to a summary as well as retain the characteristics of language model by using PPLM. The proposed method does not require any additional training. To prove the effectiveness of the proposed summarizer, this paper performed summarization experiments with Korean newspaper data.
Generative Adversarial Networks Using Pre-trained Generator for Effective Auditory Noise Suppression
http://doi.org/10.5626/JOK.2021.48.3.334
Speech enhancement GAN (SEGAN) is one of the models showing good performance in removing acoustic noise based on the genrative adversarial network, which is one of the deep learning models. However, there is a problem that the generator is easily unstable while learning non-stationary noise with a very wide distribution with one genrator. In this paper, to improve this problem, we propose an adversarial learning method using a pre-trained generator. The output of the learned generator in the same way as the autoencoder is used as the input of the adversarial learning generator. It improve stability and alleviate the difficulty of the problem, through the primary reduced noisy signal. In this paper, the Scale Invariant Signal to Noise Ratio (SI-SNR) evaluation index was used to objectively evaluate the performance of the model. As a result of the experiment, the SI-SNR increased by about 4.08 compared to the noisy speech, confirming that the proposed method is useful for removing noise.
Korean Text Summarization using MASS with Relative Position Representation
Youngjun Jung, Hyunsun Hwang, Changki Lee
http://doi.org/10.5626/JOK.2020.47.9.873
In the language generation task, deep learning-based models that generate natural languages using a Sequence-to-Sequence model are actively being studied. In the field of text summarization, wherein the method of extracting only the core sentences from the text is used, an abstract summarization study is underway. Recently, a transfer learning method of fine-tuning using pre-training model based on large amount of monolingual data such as BERT and MASS has been mainly studied in the field of natural language processing. In this paper, after pre-training for the Korean language generation using MASS, it was applied to the summarization of the Korean text. As a result of the experiment, the Korean text summarization model using MASS was higher performance than the existing models. Additionally, the performance of the text summarization model was improved by applying the relative position representation method to MASS.
Morpheme-based Korean Word Vector Generation Considering the Subword and Part-Of-Speech Information
http://doi.org/10.5626/JOK.2020.47.4.395
Word vectors enable finding the relationship between words by vector computation. They are also widely used as pre-trained data for high-level neural network programs. Various modified models from English models have been proposed for the generation of Korean word vectors, with various segmentation units such as Eojeol(word phrase), morpheme, syllable and Jaso. In this study, we propose Korean word vector generation methods that segment Eojeol into morphemes and convert them into subwords comprising either syllable or Jaso. We also propose methods using Part-Of-Speech tags provided in the pre-processing to reflect semantic and syntactic information regarding the morphemes. Intrinsic and extrinsic experiments showed that the method using morpheme segments with Jaso subwords and additional Part-Of-Speech tags showed better performance than others under the condition that the target data are normal text and not as grammatically incorrect.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr