Digital Library[ Search Result ]
A Pretrained Model-Based Approach to Improve Generalization Performance for ADMET Prediction of Drug Candidates
http://doi.org/10.5626/JOK.2025.52.7.601
Accurate prediction of Absorption, Distribution, Metabolism, Excretion, and Toxicity (ADMET) properties plays an important role in reducing clinical trial failure rates and lowering drug development costs. In this study, we propose a novel method to improve ADMET prediction performance for drug candidate compounds by integrating molecular embeddings from a graph transformer model with pretrained embeddings from a UniMol model. The proposed model can capture bond type information from molecular graph structures, generating chemically refined representations, while leveraging UniMol’s pretrained 3D embeddings to effectively learn spatial molecular characteristics. Through this, the model is designed to address the problem of data scarcity and enhance the generalization performance. In this study, we conducted prediction experiments on 10 ADMET properties. The experiment results demonstrated that our proposed model outperformed existing methods and that the prediction accuracy for ADMET properties could be improved by effectively integrating atomic bond information and 3D structures.
Generating Relation Descriptions with Large Language Model for Link Prediction
http://doi.org/10.5626/JOK.2024.51.10.908
The Knowledge Graph is a network consisting of entities and the relations between them. It is used for various natural language processing tasks. One specific task related to the Knowledge Graph is Knowledge Graph Completion, which involves reasoning with known facts in the graph and automatically inferring missing links. In order to tackle this task, studies have been conducted on both link prediction and relation prediction. Recently, there has been significant interest in a dual-encoder architecture that utilizes textual information. However, the dataset for link prediction only provides descriptions for entities, not for relations. As a result, the model heavily relies on descriptions for entities. To address this issue, we utilized a large language model called GPT-3.5-turbo to generate relation descriptions. This allows the baseline model to be trained with more comprehensive relation information. Moreover, the relation descriptions generated by our proposed method are expected to improve the performance of other language model-based link prediction models. The evaluation results for link prediction demonstrate that our proposed method outperforms the baseline model on various datasets, including Korean ConceptNet, WN18RR, FB15k-237, and YAGO3-10. Specifically, we observed improvements of 0.34%p, 0.11%p, 0.12%p, and 0.41%p in terms of Mean Reciprocal Rank (MRR), respecitvely.
Post-training Methods for Improving Korean Document Summarization Model
So-Eon Kim, Seong-Eun Hong, Gyu-Min Park, Choong Seon Hong, Seong-Bae Park
http://doi.org/10.5626/JOK.2023.50.10.882
The document summarization task generates a short summary based on a long document. Recently, a method using a pre-trained model based on a transformer model showed high performance. However, as it was proved that fine-tuning does not train the model optimally due to the learning gap between pre-training and fine-tuning, post-training, which is additional training between pre-training and fine-tuning, was proposed. This paper proposed two post-training methods for Korean document summarization. One was Korean Spacing, which is for learning Korean structure, and the other was First Sentence Masking, which is for learning about document summarization. Experiments proved that the proposed post-training methods were effective as performance improved when the proposed post-training was used compared to when it was not.
Multi-Document Summarization Use Semantic Similarity and Information Quantity of Sentence
Yeon-Soo Lim, Sunggoo Kwon, Bong-Min Kim, Seong-Bae Park
http://doi.org/10.5626/JOK.2023.50.7.561
Document summarization task has recently emerged as an important task in natural language processing because of the need for delivering concise information. However, it is difficult to obtain a suitable multi-document summarization dataset. In this paper, rather than training with a multi-document summarization dataset, we propose to use a single-document summarization dataset. That is, we propose a multi-document summarization model which generates multiple single-document summaries with a single-document summarization model and then post-processes these summaries. The proposed model consists of three modules: a summary module, a similarity module, and an information module. When multiple documents are entered into the proposed model, the summary module generates summaries of every single document. The similarity module clusters similar summaries by measuring semantic similarity. The information module selects the most informative summary from each similar summary group and collects selected summaries for the final multi-document summary. Experimental results show that the proposed model outperforms the baseline models and it can generate a high-quality multi-document summary. In addition, the performances of each module also show meaningful results.
Recommendation Technique for Bug Fixers by Fine-tuning Language Models
Dae-Sung Wang, Hoon Seong, Chan-Gun Lee
http://doi.org/10.5626/JOK.2022.49.11.987
The scale and complexity of software continue to increase; hence they contribute to the occurrence of diverse bugs. Therefore, the necessity of systematic bug management has been raised. A few studies have proposed automating the assignment of bug fixers using word-based deep learning models. However, their accuracy is not satisfactory due to context of the word is ignored, and there is an excessive number of classes. In this paper, the accuracy was improved by about 27%p over the top-10 accuracies by using a fine-tuned pre-trained language model based on BERT, RoBERTa, DeBERTa, and CodeBERT. Experiments confirmed that the accuracy was about 70%. Through this, we showed that the fine-tuned pretrained language model could be effectively applied to automated bug-fixer assignments.
Structuralized External Knowledge and Multi-task Learning for Knowledge Selection
http://doi.org/10.5626/JOK.2022.49.10.884
Typically, task-oriented dialog systems use well-structured knowledge, such as databases, to generate the most appropriate responses to users" questions. However, to generate more appropriate and fluent responses, external knowledge, which is unstructured text data such as web data or FAQs, is necessary. In this paper, we propose a novel multi-task learning method with a pre-trained language model and a graph neural network. The proposed method makes the system select the external knowledge effectively by not only understanding linguistic information but also grasping the structural information latent in external knowledge which is converted into structured data, graphs, using a dependency parser. Experimental results show that our proposed method obtains higher performance than the traditional bi-encoder or cross-encoder methods that use pre-trained language models.
Knowledge Graph Completion using Hyper-class Information and Pre-trained Language Model
http://doi.org/10.5626/JOK.2021.48.11.1228
Link prediction is a task that aims to predict missing links in knowledge graphs. Recently, several link prediction models have been proposed to complete the knowledge graphs and have achieved meaningful results. However, the previous models used only the triples" internal information in the training data, which may lead to an overfitting problem. To address this problem, we propose Hyper-class Information and Pre-trained Language Model (HIP) that performs hyper-class prediction and link prediction through a multi-task learning. HIP learns not only contextual relationship of triples but also abstractive meanings of entities. As a result, it learns general information of the entities and forces the entities connected to the same hyper-class to have similar embeddings. Experimental results show significant improvement in Hits@10 and Mean Rank (MR) compared to KG-BERT and MTL-KGC.
Topic Centric Korean Text Summarization using Attribute Model
Su-Hwan Yoon, A-Yeong Kim, Seong-Bae Park
http://doi.org/10.5626/JOK.2021.48.6.688
Abstractive summarization takes original text as an input and generates a summary containing the core-information about the original text. The abstractive summarization model is mainly designed by the Sequence-to-Sequence model. To improve quality as well as coherence of summary, the topic-centric methods which contain the core information of the original text are recently proposed. However, the previous methods perform additional training steps which make it difficult to take advantage of the pre-trained language model. This paper proposes a topic-centric summarizer that can reflect topic words to a summary as well as retain the characteristics of language model by using PPLM. The proposed method does not require any additional training. To prove the effectiveness of the proposed summarizer, this paper performed summarization experiments with Korean newspaper data.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr