Vol. 51, No. 7,
Jul. 2024
Digital Library
Accelerating DNN Models via Hierarchical N:M Sparsity
Seungmin Yu, Hayun Lee, Dongkun Shin
http://doi.org/10.5626/JOK.2024.51.7.583
N:M sparsity pruning is an effective approach for compressing deep neural networks by leveraging NVIDIA’s Sparse Tensor Core technology. Despite its effectiveness, this technique is constrained by hardware limitations, leading to fixed compression ratios and increased access to unnecessary input data, and does not adequately address the imbalanced distribution of essential parameters. This paper proposes Hierarchical N:M (HiNM) sparsity, where vector sparsity is applied prior to N:M sparsity for various-levels of sparsity. We also introduce a novel permutation technique tailored for HiNM sparsity, named 2-axis channel permutation (2CP). The experimental results showed that HiNM sparsity achieves a compression ratio twice that of traditional N:M sparsity while reducing latency by an average of 37%.
Adaptation of A Hierarchical Cumulative Prompting with Generative Large-scale Language Models in the Legal Domain
Yeenheui Yeen, HaeIn Jung, MinJu Kim, Jeong Yang, Minhye Kim, Hyunji Jang, Myoung-Wan Koo
http://doi.org/10.5626/JOK.2024.51.7.592
This study introduces a stepwise hierarchical prompting method suitable for large-scale generative language models in complex legal reasoning tasks. Complex logical problems are decomposed into multiple steps, accumulating results from each step to set prompts for subsequent ones. It was confirmed that when this method was applied to the evaluation process of the Korean bar exam's essay-type questions, it achieved better results than fine-tuning with original data. Notably, in the final evaluation by legal experts, both tasks showed a human precision of over 0.70, indicating its capability to produce interpretations based on accurate evidence. This prompting technique suggests a solution to the hallucination issue in large language models and demonstrates its effective application. Future research will consider the introduction of a specialized retriever to reflect more accurate legal knowledge in the large language model, aiming to incorporate more precise evidence into prompts. While the current research applied the prompting method only to the legal field, it is expected to be applicable to other complex logical reasoning tasks that rely on specialized knowledge.
Transformer-Based Head Motion Prediction Algorithm Using Image Generation Model
Hyogeun Byun, Moonsoo Jeong, Sungkil Lee
http://doi.org/10.5626/JOK.2024.51.7.601
Motion-to-photon latency in virtual reality based on head-mounted display can cause discomfort such as cyber sickness due to a lag between a user's physical movement and the image output, potentially disrupting users’ immersion. Traditional methods to reduce this latency involve manually analyzing head motion trends or predicting head motion with recurrent neural networks. However, these models faced long-term dependency issues in remembering information over sequences and limitations in parallel processing. In this paper, images are also used in the decoding process through an image generation model. A deep learning model used in natural language processing is highly scalable when using it as a prediction model. Accordingly, the model proposed in this study could use additional data to predict the user’s head motion and thereby, outperforms the existing models.
A Survey of Advantages of Self-Supervised Learning Models in Visual Recognition Tasks
Euihyun Yoon, Hyunjong Lee, Donggeon Kim, Joochan Park, Jinkyu Kim, Jaekoo Lee
http://doi.org/10.5626/JOK.2024.51.7.609
Recently, the field of teacher-based artificial intelligence (AI) has been rapidly advancing. However, teacher-based learning relies on datasets with specified correct answers, which can increase the cost of obtaining these correct answers. To address this issue, self-supervised learning, which can learn general features of photos without needing correct answers, is being researched. In this paper, various self-supervised learning models were classified based on their learning methods and backbone networks. Their strengths, weaknesses, and performances were then compared and analyzed. Photo classification tasks were used for performance comparison. For comparing the performance of transfer learning, detailed prediction tasks were also compared and analyzed. As a result, models that only used positive pairs achieved higher performance by minimizing noise than models that used both positive and negative pairs. Furthermore, for fine-grained predictions, methods such as masking images for learning or utilizing multi-stage models achieved higher performance by additionally learning regional information.
A Study on Development Method for BERT-based False Alarm Classification Model in Weapon System Software Static Test
Hyoju Nam, Insub Lee, Namhoon Jung, Seongyun Jeong, Kyutae Cho, Sungkyu Noh
http://doi.org/10.5626/JOK.2024.51.7.620
Recently, as the size and complexity of software in weapon systems have increased, securing the reliability and stability is required. To achieve this, developers perform static and dynamic reliability testing during development. However, a lot of false alarms occur in static testing progress that cause wasting resources such as time and cost for reconsider them. Recent studies have tried to solve this problem by using models such as SVM and LSTM. However, they have a critical limitation in that these models do not reflect correlation between defect code line and other lines since they use Word2Vec-based code embedding or only code information. The BERT-based model learns the front-to-back relationship between sentences through the application of a bidirectional transformer. Therefore, it can be used to classify false alarms by analyzing the relationship between code. In this paper, we proposed a method for developing a false alarm classification model using a BERT-based model to efficiently analyze static test results. We demonstrated the ability of the proposed method to generate a dataset in a development environment and showed the superiority of our model.
Improving Prediction of Chronic Hepatitis B Treatment Response Using Molecular Embedding
Jihyeon Song, Soon Sun Kim, Ji Eun Han, Hyo Jung Cho, Jae Youn Cheong, Charmgil Hong
http://doi.org/10.5626/JOK.2024.51.7.627
Chronic hepatitis B patients with no timely treatment are at a high risk of developing complications such as liver cirrhosis and hepatocellular carcinoma (liver cancer). As a result, various antiviral agents for hepatitis B have been developed, and due to the different components of these antiviral agents, there can be variations in treatment responses among patients. Therefore, selecting the appropriate medication that leads to a favorable treatment response is considered crucial. In this study, in addition to the patient's blood test results and electronic medical records indicating drug prescriptions, information about components of the hepatitis B antiviral agents was incorporated for learning. The aim was to enhance the prediction performance of treatment responses one year after chronic hepatitis B patients' treatment. Molecular embedding of the antiviral agents included both fixed molecular embedding and those generated through an end-to-end structure utilizing a graph neural network model. By comparing with the baseline model, drug molecule embedding was confirmed to contribute to improving performance.
Predicting of the Number of Diners in School Cafeteria; Including COVID-19 Pandemic Period Data
Chae-eun Baek, Yesl Kwon, Jangmin Oh
http://doi.org/10.5626/JOK.2024.51.7.634
Accurately predicting the number of diners in institutional food service is essential for efficient operations, reducing leftovers, and ensuring customer satisfaction. University cafeterias, in particular, face additional challenges in making these predictions due to various environmental factors and changes in class formats caused by the COVID-19 pandemic. To tackle this issue, this study utilized specialized data collected during the pandemic period in university cafeteria environments. The data was used to train and compare the performance of five different models. The three best-performing ensemble tree-based models -- RandomForest, LightGBM, and XGBoost -- were averaged to obtain a final prediction with a Mean Absolute Error (MAE) of 30.96. By regularly providing prediction results to on-campus cafeterias using this final model, practical support can be offered to optimize operations. This study presents an effective methodology for accurately predicting of the number of diners, even in abnormal situations such as the COVID-19 pandemic.
Memory Model Design for Integer-Pointer Casting Support in C-like languages Via Dual Non-determinism
http://doi.org/10.5626/JOK.2024.51.7.643
In system programming, pointers are essential elements. However, applying formal verification methods to programs involving integer-pointer casting poses an important challenge. To address this challenge, a mathematically defined memory model that supports integer-pointer casting, along with proof techniques for verification, is necessary. This study presents a memory model that supports integer-pointer casting within the Coq proof assistant. The model accommodates patterns associated with integer-pointer operations, including one-past-the-end pointers. Additionally, a simulation-based proof technique is introduced, which enables the utilization of the model for program verification. The adequacy of this technique is established through proof. To validate the effectiveness of the approach, the defined memory model is integrated into CompCert, a verified C compiler, replacing its original memory model. Subsequently, two proofs of CompCert's optimization verification are updated using the simulation technique. It is anticipated that the proposed memory model will find applications in program and compiler verification tasks involving integer-pointer operations.
Efficient Prompt Learning Method in Blurry Class Incremental Learning Environment
http://doi.org/10.5626/JOK.2024.51.7.655
Continual learning is the process of continuously integrating new knowledge to maintain performance across a sequence of tasks. While disjoint continual learning, which assumes no overlap between classes across tasks, blurry continual learning addresses more realistic scenarios where overlaps do exist. Traditionally, most related works have predominantly focused on disjoint scenarios and recent attention has shifted towards prompt-based continual learning. This approach uses prompt mechanism within a Vision Transformer (ViT) model to improve adaptability. In this study, we analyze the effectiveness of a similarity function designed for blurry class incremental learning, applied within a prompt-based continual learning framework. Our experiments demonstrate the success of this method, particularly in its superior ability to learn from and interpret blurry data.
A GRU-based Time-Series Forecasting Method using Patching
http://doi.org/10.5626/JOK.2024.51.7.663
Time series forecasting plays a crucial role in decision-making within various fields. Two recent approaches, namely, the patch time series Transformer (PatchTST) and the long-term time series foraging linear (LTSF-Linear) of the MLP structure have shown promising performance in this area. However, PatchTST requires significant time for both model training and inference, while LTSF-Linear has limited capacity due to its simplistic structure. To address these limitations, we propose a new approach called patch time series GRU (PatchTSG). By leveraging a Gated Recurrent Unit (GRU) on the patched data, PatchTSG reduces the training time and captures valuable information from the time series data. Compared to PatchTST, PatchTSG achieves an impressive reduction in learning time (up to 82%) and inference time (up to 46%).
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr