Digital Library[ Search Result ]
A Pretrained Model-Based Approach to Improve Generalization Performance for ADMET Prediction of Drug Candidates
http://doi.org/10.5626/JOK.2025.52.7.601
Accurate prediction of Absorption, Distribution, Metabolism, Excretion, and Toxicity (ADMET) properties plays an important role in reducing clinical trial failure rates and lowering drug development costs. In this study, we propose a novel method to improve ADMET prediction performance for drug candidate compounds by integrating molecular embeddings from a graph transformer model with pretrained embeddings from a UniMol model. The proposed model can capture bond type information from molecular graph structures, generating chemically refined representations, while leveraging UniMol’s pretrained 3D embeddings to effectively learn spatial molecular characteristics. Through this, the model is designed to address the problem of data scarcity and enhance the generalization performance. In this study, we conducted prediction experiments on 10 ADMET properties. The experiment results demonstrated that our proposed model outperformed existing methods and that the prediction accuracy for ADMET properties could be improved by effectively integrating atomic bond information and 3D structures.
Improved Software Defect Prediction with Gated Tab Transformer
Saranya Manikandan, Duksan Ryu
http://doi.org/10.5626/JOK.2025.52.3.196
Software Defect Prediction (SDP) plays a crucial role in ensuring software quality and reliability. Although, traditional machine learning and deep learning models are widely used for SDP, recent advancements in the field of natural language processing have paved the way for applying transformer-based models in software engineering tasks. This paper investigated transformer-based model as a potential approach to improve SDP model quality, ultimately aiming to enhance software quality and optimize testing resource allocation. Inspired by the Gated Tab Transformer’s (GTT) ability to effectively model relationship within features, we evaluated its effectiveness in SDP. We conducted experiments using 15 software defect datasets and compared results with other state-of-the-art machine learning and deep learning models. Our experiments showed that GTT outperformed state-of-the-art machine learning models in terms of recall, balance, and AUC (increase by 42.1%, 10.93%, and 7.1%, respectively). Cohen's d confirmed this advantage with large and medium effect sizes for GTT on these metrics. Additionally, an ablation study assessed the impact of hyperparameter variations on performance. Thus, GTT's effectiveness address the challenges of SDP, potentially leading to more effective testing resource allocation and improved software quality.
Maximizing UAV Data Efficiency in NextG Networks: A Transformer-Based mmWave Beamforming Approach
Avi Deb Raha, Apurba Adhikary, Mrityunjoy Gain, Yu Qiao, Hyeonsu Kim, Jisu Yoon, Choong Seon Hong
http://doi.org/10.5626/JOK.2025.52.2.170
Beamforming is essential in the rapidly evolving field of next generation (NextG) wireless communication, particularly when leveraging terahertz and millimeter-wave (mmWave) frequency bands to achieve ultra-high data speeds. However, these frequency bands present challenges, particularly concerning the costs associated with beam training, which can hinder Ultra-Reliable Low-Latency Communication (URLLC) in high-mobility applications, such as drone and Unmanned Aerial Vehicle (UAV) communications. This paper proposes a contextual information-based mmWave beamforming approach for UAVs and formulates an optimization problem aimed at maximizing data rates in high-mobility UAV scenarios. To predict optimal beams while ensuring URLLC, we have developed a lightweight transformerThe self-attention mechanism of the transformer allows the model to focus selectively on the most important features of the contextual information. This lightweight transformer model effectively predicts the best beams, thereby enhancing the data rates of UAVs. Simulation results demonstrate the design's effectiveness, as the lightweight transformer model significantly outperforms baseline methods, achieving up to 17.8% higher Top-1 beam accuracies and reducing average power loss by as much as 96.79%. Improvements range from 12.49% to 96.79% relative to baseline methods.
Topic-Aware Cross-Attention for Dialogue Summarization
http://doi.org/10.5626/JOK.2024.51.11.1011
Unlike general document summarization, dialogue summarization frequently involves informal and colloquial language. It requires an understanding of the context and flow of the dialogue. It also requires consideration of topics. This study proposes a Topic-Aware Cross-Attention mechanism that can incorporate elements to recognize topic distributions into a cross-attention mechanism to reflect characteristics of dialogue. This Topic-Aware Cross-Attention mechanism can extract topic distributions of dialogue and summary and apply the similarity of these distributions to the cross-attention mechanism within BART model’s decoder to perform dialogue summarization. The proposed Topic-Aware Cross-Attention mechanism can adjust application degree of topic distribution similarity to the cross-attention mechanism by modifying topic-ratio. Experimental results on DialogSum and SAMSum datasets demonstrated the suitability of the method for dialogue summarization.
A Survey of Advantages of Self-Supervised Learning Models in Visual Recognition Tasks
Euihyun Yoon, Hyunjong Lee, Donggeon Kim, Joochan Park, Jinkyu Kim, Jaekoo Lee
http://doi.org/10.5626/JOK.2024.51.7.609
Recently, the field of teacher-based artificial intelligence (AI) has been rapidly advancing. However, teacher-based learning relies on datasets with specified correct answers, which can increase the cost of obtaining these correct answers. To address this issue, self-supervised learning, which can learn general features of photos without needing correct answers, is being researched. In this paper, various self-supervised learning models were classified based on their learning methods and backbone networks. Their strengths, weaknesses, and performances were then compared and analyzed. Photo classification tasks were used for performance comparison. For comparing the performance of transfer learning, detailed prediction tasks were also compared and analyzed. As a result, models that only used positive pairs achieved higher performance by minimizing noise than models that used both positive and negative pairs. Furthermore, for fine-grained predictions, methods such as masking images for learning or utilizing multi-stage models achieved higher performance by additionally learning regional information.
Efficient Prompt Learning Method in Blurry Class Incremental Learning Environment
http://doi.org/10.5626/JOK.2024.51.7.655
Continual learning is the process of continuously integrating new knowledge to maintain performance across a sequence of tasks. While disjoint continual learning, which assumes no overlap between classes across tasks, blurry continual learning addresses more realistic scenarios where overlaps do exist. Traditionally, most related works have predominantly focused on disjoint scenarios and recent attention has shifted towards prompt-based continual learning. This approach uses prompt mechanism within a Vision Transformer (ViT) model to improve adaptability. In this study, we analyze the effectiveness of a similarity function designed for blurry class incremental learning, applied within a prompt-based continual learning framework. Our experiments demonstrate the success of this method, particularly in its superior ability to learn from and interpret blurry data.
A GRU-based Time-Series Forecasting Method using Patching
http://doi.org/10.5626/JOK.2024.51.7.663
Time series forecasting plays a crucial role in decision-making within various fields. Two recent approaches, namely, the patch time series Transformer (PatchTST) and the long-term time series foraging linear (LTSF-Linear) of the MLP structure have shown promising performance in this area. However, PatchTST requires significant time for both model training and inference, while LTSF-Linear has limited capacity due to its simplistic structure. To address these limitations, we propose a new approach called patch time series GRU (PatchTSG). By leveraging a Gated Recurrent Unit (GRU) on the patched data, PatchTSG reduces the training time and captures valuable information from the time series data. Compared to PatchTST, PatchTSG achieves an impressive reduction in learning time (up to 82%) and inference time (up to 46%).
A Token Selection Method for Effective Token Pruning in Vision Transformers
http://doi.org/10.5626/JOK.2024.51.6.567
The self-attention-based models, vision transformers, have recently been employed in the field of computer vision. While achieving excellent performance in a variety of tasks, the computation costs increase in proportion to the number of tokens during inference, which causes a degradation in inference speed. Especially when deploying the model in real-world scenarios, many limitations could be encountered. To address this issue, we propose a new token importance measurement, which can be obtained by modifying the structure of multi-head self-attention in vision transformers. By pruning less important tokens through our method during inference, we can improve inference speed while preserving performance. Furthermore, our proposed method, which requires no additional parameters, exhibits better robustness without fine-tuning and demonstrates that it can maximize performance when integrated with existing token pruning methods.
ConTL: Improving the Performance of EEG-based Emotion Recognition via the Incorporation of CNN, Transformer and LSTM
Hyunwook Kang, Byung Hyung Kim
http://doi.org/10.5626/JOK.2024.51.5.454
This paper proposes a hybrid-network called ConTL, which is composed of a convolutional neural network (CNN), Transformer, and long short-term memory (LSTM) for EEG-based emotion recognition. Firstly, CNN is exploited to learn local features from the input EEG signals. Then, the Transformer learns global temporal dependencies from the output features. To further learn sequential dependencies of the time domain, the output features from the Transformer are fed to the bi-directional LSTM. To verify the effects of the proposed model, we compared the classification accuracies with five state-of-the-art models. There was an 0.73% improvement on SEED-IV compared to CCNN, and improvements of 0.97% and 0.63% were observed compared to DGCNN for valence and arousal of DEAP, respectively.
SCA: Improving Document Grounded Response Generation based on Supervised Cross-Attention
Hyeongjun Choi, Seung-Hoon Na, Beomseok Hong, Youngsub Han, Byoung-Ki Jeon
http://doi.org/10.5626/JOK.2024.51.4.326
Document-grounded response generation is the task of aiming at generating conversational responses by “grounding” the factual evidence on task-specific domain, such as consumer consultation or insurance planning, where the evidence is obtained from the retrieved relevant documents in response to a user’s question under the current dialogue context. In this study, we propose supervised cross-attention (SCA) to enhance the ability of the response generation model to find and incorporate “response-salient snippets” (i.e., spans or contents), which are parts of the retrieved document that should be included and maintained in the actual answer generation. SCA utilizes the additional supervised loss that focuses cross-attention weights on the response-salient snippets, and this attention supervision likely enables a decoder to effectively generate a response in a “saliency-grounding” manner, by strongly attending to the important parts in the retrieved document. Experiment results on MultiDoc2Dial show that the use of SCA and additional performance improvement methods leads to the increase of 1.13 in F1 metric over the existing SOTA, and reveals that SCA leads to the increase of 0.25 in F1.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr