Digital Library[ Search Result ]
A Token Selection Method for Effective Token Pruning in Vision Transformers
http://doi.org/10.5626/JOK.2024.51.6.567
The self-attention-based models, vision transformers, have recently been employed in the field of computer vision. While achieving excellent performance in a variety of tasks, the computation costs increase in proportion to the number of tokens during inference, which causes a degradation in inference speed. Especially when deploying the model in real-world scenarios, many limitations could be encountered. To address this issue, we propose a new token importance measurement, which can be obtained by modifying the structure of multi-head self-attention in vision transformers. By pruning less important tokens through our method during inference, we can improve inference speed while preserving performance. Furthermore, our proposed method, which requires no additional parameters, exhibits better robustness without fine-tuning and demonstrates that it can maximize performance when integrated with existing token pruning methods.
Improvement of Deep Learning Models to Predict the Knowledge Level of Learners based on the EdNet Data
Seulgi Choi, Youngpyo Kim, Sojung Hwang, Heeyoul Choi
http://doi.org/10.5626/JOK.2021.48.12.1335
As online education increases, the field of AI in Education (AIEd), where artificial intelligence is used for education, is being actively studied. Knowledge Tracing (KT), which predicts a student"s knowledge level based on each student"s learning record, is a basic task in the AIEd field. However, there is a lack of utilization of the dataset and research on the KT model architecture. In this paper, we propose to use a total of 11 features, after trying various features related to the problems, and present a new model based on the self-attention mechanism with new query, key, and values, Self-Attentive Knowledge Tracking Extended (SANTE). In experiments, we confirm that the proposed method with the selected features outperforms the previous KT models in terms of the AUC value.
Low-Resolution Image Classification Using Knowledge Distillation From High-Resolution Image Via Self-Attention Map
Sungho Shin, Joosoon Lee, Junseok Lee, Seungjun Choi, Kyoobin Lee
http://doi.org/10.5626/JOK.2020.47.11.1027
Traditional deep-learning models have been developed using high-quality images. However, when the low resolution images are rendered, the performances of the model drop drastically. To develop a deep-learning model that can respond effectively to low-resolution images, we extracted the information from the model, which uses high-resolution images as input, in the form of the Attention Map. Using the knowledge distillation technique, the information delivering Attention Map, extracted from the high-resolution images to low-resolution image models, could reduce the error rate by 2.94%, when classifying the low-resolution CIFAR images of 16×16 resolution. This was at 38.43% of the error reduction rate when the image resolution was lowered from 32×32 to 16×16, which could demonstrate excellence in this network.
Korean Movie Review Sentiment Analysis using Self-Attention and Contextualized Embedding
Cheoneum Park, Dongheon Lee, Kihoon Kim, Changki Lee, Hyunki Kim
http://doi.org/10.5626/JOK.2019.46.9.901
Sentiment analysis is the processing task that involves collecting and classifying opinions about a specific object. However, it is difficult to grasp the subjectivity of a person using natural language, so the existing sentimental word dictionaries or probabilistic models cannot solve such a task, but the development of deep learning made it possible to solve the task. Self-attention is a method of modeling a given input sequence by calculating the attention weight of the input sequence itself and constructing a context vector with a weighted sum. In the context, a high weight is calculated between words with similar meanings. In this paper, we propose a method using a modeling network with self-attention and pre-trained contextualized embedding to solve the sentiment analysis task. The experimental result shows an accuracy of 89.82%.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr