A Retrieval Augmented Generation(RAG) System Using Query Rewritting Based on Large Langauge Model(LLM)

Minsu Han, Seokyoung Hong, Myoung-Wan Koo

http://doi.org/10.5626/JOK.2025.52.6.474

This paper proposes a retrieval pipeline that can be effectively utilized in fields requiring expert knowledge without requiring fine-tuning. To achieve high accuracy, we introduce a query rewriting retrieval method that leverages large language models to generate examples similar to the given question, achieving higher similarity than existing retrieval models. The proposed method demonstrates excellent performance in both automated evaluations and expert qualitative assessments, while also providing explainability in retrieval results through generated examples. Additionally, we suggest prompts that can be utilized in various domains requiring specialized knowledge during the application of this method. Furthermore, we propose a pipeline method that incorporates a Top-1 retrieval model, which chooses the most relevant document from the three returned by the query rewriting retrieval model. This aims to prevent the hallucination issue caused by the input of unnecessary documents into the large language model.

Pretrained Large Language Model-based Drug-Target Binding Affinity Prediction for Mutated Proteins

Taeung Song, Jin Hyuk Kim, Hyeon Jun Park, Jonghwan Choi

http://doi.org/10.5626/JOK.2025.52.6.539

Drug development is a costly and time-consuming process. Accurately predicting the impact of protein mutations on drug-target binding affinity remains a major challenge. Previous studies have utilized long short-term memory (LSTM) and transformer models for amino acid sequence processing. However, LSTMs suffer from long-sequence dependency issues, while transformers face high computational costs. In contrast, pretrained large language models (pLLMs) excel in handling long sequences, yet prompt-based approaches alone are insufficient for accurate binding affinity prediction. This study proposed a method that could leverage pLLMs to analyze protein structural data, transform it into embedding vectors, and use a separate machine learning model for numerical binding affinity prediction. Experimental results demonstrated that the proposed approach outperformed conventional LSTM and prompt-based methods, achieving lower root mean square error (RMSE) and higher Pearson correlation coefficient (PCC), particularly in mutation-specific predictions. Additionally, performance analysis of pLLM quantization confirmed that the method maintained sufficient accuracy with reduced computational cost.

A Large Language Model-based Multi-domain Recommender System using Model Merging

Hyunsoo Kim, Jongwuk Lee

http://doi.org/10.5626/JOK.2025.52.6.548

Recent research in recommender systems has increasingly focused on leveraging pre-trained large language models (LLMs) to effectively understand the natural language information associated with recommendation items. While these LLM-based recommender systems achieve high accuracy, they have a limitation in that they require training separate recommendation models for each domain. This increases the costs of storing and inferring multiple models and makes it difficult to share knowledge across domains. To address this issue, we propose an LLM-based recommendation model that effectively operates across diverse recommendation domains by applying task vector-based model merging. During the merging process, knowledge distillation is utilized from individually trained domain-specific recommendation models to learn optimal merging weights. Experimental results show that our proposed method improves recommendation accuracy by an average of 2.75% across eight domains compared to recommender models utilizing existing model merging methods, while also demonstrating strong generalization performance in previously unseen domains.

An Inference Framework for Text-Based Sequential Recommendation Model Using Nearest Neighbor Mechanism

Junyoung Kim, Hyunsoo, Jongwuk Lee

http://doi.org/10.5626/JOK.2025.52.5.435

Sequential recommendation task aims to predict the next item to interact with based on users’ interaction history. Text-based recommendation models, which represent items as text, show improved performance in cold-start problems and zero-shot recommendation tasks. However, they suffer from textual bias and the lack of collaborative knowledge. To overcome these limitations, we propose a text-based recommendation model inference framework using the nearest neighbor mechanism. The proposed method leverages text-based recommendation models as a neighbor retriever model to search neighbors with similar preferences to the user and aggregate the neighbor information with existing recommendation results to improve recommendation performance. Experiments conducted on four datasets show that the proposed method consistently outperforms existing models, with performance improvement up to 25.27% on NDCG@50. Furthermore, the proposed method effectively complements collaborative knowledge and improves model explainability by providing recommendation rationale.

Hallucination Detection and Explanation Model for Enhancing the Reliability of LLM Responses

Sujeong Lee, Hayoung Lee, Seongsoo Heo, Wonik Choi

http://doi.org/10.5626/JOK.2025.52.5.404

Recent advancements in large language models (LLMs) have achieved remarkable progress in natural language processing. However, reliability issues persist due to hallucination, which remains a significant challenge. Existing hallucination research primarily focuses on detection, lacking the capability to explain the causes and context of hallucinations. In response, this study proposes a hallucination-specialized model that goes beyond mere detection by providing explanations for identified hallucinations. The proposed model was designed to classify hallucinations while simultaneously generating explanations, allowing users to better trust and understand the model’s responses. Experimental results demonstrated that the proposed model surpassed large-scale models such as Llama3 70B and GPT-4 in hallucination detection accuracy while consistently generating high-quality explanations. Notably, the model maintained stable detection and explanation performance across diverse datasets, showcasing its adaptability. By integrating hallucination detection with explanation generation, this study introduces a novel approach to evaluating hallucinations in language models.

Enhancing Passage Selection and Answer Generation in FiD Systems Using Relevance Gating

Seung-ho Choi, Shihyun Park, Minsang Kim, Chansol Park, Junho Wang, Ji-Yoon Kim, Bong-Su Kim

http://doi.org/10.5626/JOK.2025.52.5.385

In this paper, we proposed a novel approach to enhance the performance of the Fusion-in-Decoder (FiD) model in open-domain question answering systems. The FiD model operates by independently encoding multiple passages and then combining them during the decoding stage to generate answers. However, this method has the drawback of not filtering out passages containing unnecessary information, thereby placing an excessive burden on the decoder. To address this issue, we introduced a Relevance Gate inspired by the forget gate of Long Short-Term Memory (LSTM). This gate can evaluate the relevance of each passage in parallel, selectively transmitting information to the decoder, thereby significantly improving the accuracy and efficiency of answer generation. Additionally, we applied a new activation function suitable for open-domain question answering systems instead of the sigmoid function to ensure the model's stability.

Enhancing Molecular Understanding in LLMs through Multimodal Graph-SMILES Representations

http://doi.org/10.5626/JOK.2025.52.5.379

Recent advancements in large language models (LLMs) have shown remarkable performace across various tasks, with increasing focus on multimodal research. Notably, BLIP-2 can enhance performance by efficiently aligning images and text using a Q-Former, aided by an image encoder pre-trained on multimodal data. Inspired by this, the MolCA model extends BLIP-2 to the molecular domain to improve performance. However, the graph encoder in MolCA is pre-trained on unimodal data, necessitating updates during model training, which is a limitation. Therefore, this paper replaced it with a graph encoder pre-trained on multimodal data and frozen while training the model. Experimental results showed that using the graph encoder pre-trained on multimodal data generally enhanced performance. Additionally, unlike the graph encoder pre-trained on unimodal data, which performed better when updated, the graph encoder pre-trained on multimodal data achieved superior results across all metrics when frozen.

Enhancing Retrieval-Augmented Generation Through Zero-Shot Sentence-Level Passage Refinement with LLMs

Taeho Hwang, Soyeong Jeong, Sukmin Cho, Jong C. Park

http://doi.org/10.5626/JOK.2025.52.4.304

This study presents a novel methodology designed to enhance the performance and effectiveness of Retrieval-Augmented Generation (RAG) by utilizing Large Language Models (LLMs) to eliminate irrelevant content at the sentence level from retrieved documents. This approach refines the content of passages exclusively through LLMs, avoiding the need for additional training or data, with the goal of improving the performance in knowledge-intensive tasks. The proposed method was tested in an open-domain question answering (QA) environment, where it demonstrated its ability to effectively remove unnecessary content and outperform over traditional RAG methods. Overall, our approach has proven effective in enhancing performance compared to conventional RAG techniques and has shown the capability to improve RAG's accuracy in a zero-shot setting without requiring additional training data.

Efficiently Lightweight Korean Language Model with Post-layer Pruning and Multi-stage Fine-tuning

Jae Seong Kim, Suan Lee

http://doi.org/10.5626/JOK.2025.52.3.260

The increasing size of large-scale language models has led to the need for lightweighting for practical applications. This study presents a method to reduce the existing 8B model to 5B by late-layer pruning, while maintaining and improving its performance through two phases of fine-tuning. In the broad fine-tuning phase, we expanded the model's ability to understand and generate Korean by utilizing English-Korean parallel data and a large Korean corpus, and in the refined fine-tuning phase, we enhanced its expressive and inferential capabilities with high-quality datasets. In addition, we integrated the strengths of individual models through model merging techniques. In the LogicKor leaderboard evaluation, the proposed model performed well in the areas of reasoning, writing, and comprehension, with an overall score of 4.36, outperforming the original Llama-3.1-8B-Instruct model (4.35). This demonstrates a 37.5% reduction in model size while still improving performance.

SyllaBERT: A Syllable-Based Efficient Robust Transformer Model for Real-World Noise and Typographical Errors

Seongwan Park, Yumin Heo, Youngjoong Ko

http://doi.org/10.5626/JOK.2025.52.3.250

Training a Korean language model necessitates the development of a tokenizer specifically designed for the unique features of the Korean language, making this a crucial step in the modeling process. Most current language models utilize morpheme-based or subword-based tokenization. While these approaches work well with clean Korean text data, they are prone to out-of-vocabulary (OOV) issues due to abbreviations and neologisms frequently encountered in real-world Korean data. Moreover, actual Korean text often contains various typos and non-standard expressions, to which traditional morpheme-based or subword-based tokenizers are not sufficiently robust. To tackle these challenges, this paper introduces the SyllaBERT model, which employs syllable-level tokenization to effectively address the specific characteristics of Korean, even in noisy and non-standard contexts, with minimal resources. A compact syllable-level vocabulary was created, and a syllable-based language model was developed by reducing the embedding and hidden layer sizes of existing models. Experimental results show that, despite having approximately four times fewer parameters than subword-based models, the SyllaBERT model outperforms them in natural language understanding tasks on real-world conversational Korean data that includes noise.