Efficient Large Language Model Based Passage Re-Ranking Using Single Token Representations

Jeongwoo Na, Jun Kwon, Eunseong Choi, Jongwuk Lee

http://doi.org/10.5626/JOK.2025.52.5.395

In information retrieval systems, document re-ranking involves reordering a set of candidate documents based on evaluation of their relevance to a given query. Leveraging extensive natural language understanding capabilities of large language models(LLMs), numerous studies on document re-ranking have been conducted, demonstrating groundbreaking performance. However, studies utilizing large language models focus solely on improving reranking performance, resulting in degraded efficiency due to excessively long input sequences and the need for repetitive inference. To address these limitations, we propose ListT5++, a novel model that represents the relevance between a query and a passage using single token embedding and significantly improves the efficiency of LLM-based reranking through a single-step decoding strategy that minimizes the decoding process. Experimental results showed that ListT5++ could maintain accuracy levels comparable to existing methods while reducing inference latency by a factor of 29.4 relative to the baseline. Moreover, our approach demonstrates robust characteristics by being insensitive to th initial ordering of candidate documents, thereby ensuring high practicality in real-time retrieval environments.

Enhancing Molecular Understanding in LLMs through Multimodal Graph-SMILES Representations

http://doi.org/10.5626/JOK.2025.52.5.379

Recent advancements in large language models (LLMs) have shown remarkable performace across various tasks, with increasing focus on multimodal research. Notably, BLIP-2 can enhance performance by efficiently aligning images and text using a Q-Former, aided by an image encoder pre-trained on multimodal data. Inspired by this, the MolCA model extends BLIP-2 to the molecular domain to improve performance. However, the graph encoder in MolCA is pre-trained on unimodal data, necessitating updates during model training, which is a limitation. Therefore, this paper replaced it with a graph encoder pre-trained on multimodal data and frozen while training the model. Experimental results showed that using the graph encoder pre-trained on multimodal data generally enhanced performance. Additionally, unlike the graph encoder pre-trained on unimodal data, which performed better when updated, the graph encoder pre-trained on multimodal data achieved superior results across all metrics when frozen.

Improving the Performance of Knowledge Tracing Models using Quantized Correctness Embeddings

Yoonjin Im, Jaewan Moon, Eunseong Choi, Jongwuk Lee

http://doi.org/10.5626/JOK.2023.50.4.329

Knowledge tracing is a task of monitoring the proficiency of knowledge based on learners" interaction records. Despite the flexible usage of deep neural network-based models for this task, the existing methods disregard the difficulty of each question and result in poor performance for learners who get the easy question wrong or the hard question correct. In this paper, we propose quantizing the learners’ response information based on the question difficulty so that the knowledge tracing models can learn both the response and the difficulty of the question in order to improve the performance. We design a method that can effectively discriminate between negative samples with a high percentage of correct answer rate and positive samples with a low percentage of correct answer rate. Toward this end, we use sinusoidal positional encoding (SPE) that can maximize the distance difference between embedding representations in the latent space. Experiments show that the AUC value is improved to a maximum of 17.89% in the target section compared to the existing method.

Data Augmentation Methods for Improving the Performance of Machine Reading Comprehension

Sunkyung Lee, Eunseong Choi, Seonho Jeong, Jongwuk Lee

http://doi.org/10.5626/JOK.2021.48.12.1298

Machine reading comprehension is a method of understanding the meaning and performing inference over a given text by computers, and it is one of the most essential techniques for understanding natural language. The question answering task yields a way to test the reasoning ability of intelligent systems. Nowadays, machine reading comprehension techniques performance has significantly improved following the recent progress of deep neural networks. Nevertheless, there may be challenges in improving performance when data is sparse. To address this issue, we leverage word-level and sentence-level data augmentation techniques through text editing, while minimizing changes to the existing models and cost. In this work, we propose data augmentation methods for a pre-trained language model, which is most widely used in English question answering tasks, to confirm the improved performance over the existing models.