Vol. 48, No. 4,
Apr. 2021
Digital Library
Conditional Knowledge Distillation for Model Specialization
http://doi.org/10.5626/JOK.2021.48.4.369
Many recent works on model compression in neural networks are based on knowledge distillation (KD). However, since the basic goal of KD is to transfer the entire knowledge set of a teacher model to a student model, the standard KD may not represent the best use of the model’s capacity when a user wishes to classify only a small subset of classes. Also, it is necessary to possess the original teacher model dataset for KD, but for various practical reasons, such as privacy issues, the entire dataset may not be available. Thus, this paper proposes conditional knowledge distillation (CKD), which only distills specialized knowledge corresponding to a given subset of classes, as well as data-free CKD (DF-CKD), which does not require the original data. As a major extension, we devise Joint-CKD, which jointly performs DF-CKD and CKD with only a small additional dataset collected by a client. Our experimental results show that the CKD and DF-CKD methods are superior to standard KD, and also confirm that joint use of CKD and DF-CKD is effective at further improving the overall accuracy of a specialized model.
Computational Analysis of Heuristics and Linear Programming Relaxations for Capacitated Facility Location
http://doi.org/10.5626/JOK.2021.48.4.377
In this paper, we experimentally evaluate new practical heuristics based on approximation algorithms for capacitated facility location (CFL) and the linear programming (LP) relaxations used by them. Although CFL has been extensively studied in the fields of computer science and operations research, an LP-based constant-factor approximation algorithm was only discovered recently. However, the existing analysis only gives a loose upper bound of 288 on the integrality gap of the LP relaxation. By measuring the approximation ratio and the integrality gap in a concrete dataset, this paper aims at measuring the practical performance of the heuristics and the relaxation, which would also provide evidence for further theoretical exploration. Due to the highly theoretical nature of the existing approximation algorithm, its naive implementation is practically inefficient. In this paper, we propose heuristics that improve the running time without affecting the numerical stability and experimentally optimize the algorithms’ parameters, thus obtaining an efficient implementation.
Dynamic Function Relevance based Fuzzing for High Coverage
Ahcheong Lee, Yunho Kim, Moonzoo Kim
http://doi.org/10.5626/JOK.2021.48.4.391
Coverage Guided Fuzzing (CGF) is one of the famous test case generation technique. The technique is actively researched and used based on its simplicity and scalability for large real software. However, most of the fuzzing techniques do not utilize valuable semantic information of target programs. This paper presents two new heuristics that use dynamic function relevance to select the appropriate input bytes which can be mutated to increase the coverage. The function relevance between the two functions is defined as the number of test cases that execute the functions together, and the high relevance means the two functions executing high dependency on each other. To improve coverage of a target function, the new heuristics determines bytes that are used by functions that are highly relevant to the target function, and only the valuable bytes are mutated. As these bytes have high data dependency on the variables in the target function, mutating them improves the coverage of the target function. We implemented the two heuristics on the top of the state-of-the-art fuzzers, Angora and FairFuzz, and evaluated on real-world C programs that are used by recent fuzzing works. The heuristics showed 17.88% and 11.03% path coverage improvement, respectively.
Facial Emotion Recognition Data Augmentation using Generative Adversarial Network
http://doi.org/10.5626/JOK.2021.48.4.398
The facial emotion recognition field of computer vision has recently been identified to demonstrate meaningful results through various neural networks. However, the major datasets of facial emotion recognition have the problem of “class imbalance,” which is a factor that degrades the accuracy of deep learning models. Therefore, numerous studies have been actively conducted to solve the problem of class imbalance. In this paper, we propose “RDGAN,” a facial emotion recognition data augmentation model that uses a GAN to solve the class imbalance of the FER2013 and RAF_single that are used as facial emotion recognition datasets. RDGAN is a network that generates images suitable for classes by adding expression discriminators based on the image-to-image translation model between the existing images as compared to the prevailing studies. The dataset augmented with RDGAN showed an average performance improvement of 4.805%p and 0.857%p in FER2013 and RAF_single, respectively, compared to the dataset without data augmentation.
ConvLSTM-Based COVID-19 Outbreak Prediction using Feature Combination of Multivariate Dataset
Yejin Kim, Seokyeon Kim, Yun Jang
http://doi.org/10.5626/JOK.2021.48.4.405
COVID-19 is transmitted through the droplets expelled by infected people. The propagation of splash is affected by space-time. The transmission of infectious diseases depends on the interaction of various factors such as the health status of the infected and the non-infected people and different environmental factors. However, it is difficult to include all information related to the epidemic in the predictive model and understand the relationship between the information. In this research, we propose a method to include the infectious features of COVID-19 in a learning dataset of the deep learning model and understand the effect of the combination of COVID-19 spreading data on the predictive performance of deep learning. Before predicting, the infectious features of COVID-19 are identified and considerations for including the COVID-19 spreading features are defined in the data preprocessing step. In deep learning modeling, a prediction model using ConvLSTM is designed for spatiotemporal prediction. In the process of testing the model, various features related to COVID-19 spread are combined and the effect of the combination on the performance of the model is analyzed. We tested 120 feature combinations with 47 features composed of personal information of confirmed patients and spatial characteristics of the places that they had visited. We used MAPE as an indicator to evaluate performance of the models. In the case of COVID-19 dataset, the MAPE value of the model with combined features was 1.234, and that of the model with not combined features was 2.217.
Improvement in Network Intrusion Detection based on LSTM and Feature Embedding
Hyeokmin Gwon, Chungjun Lee, Rakun Keum, Heeyoul Choi
http://doi.org/10.5626/JOK.2021.48.4.418
Network Intrusion Detection System (NIDS) is an essential tool for network perimeter security. NIDS inspects network traffic packets to detect network intrusions. Most of the existing works have used machine learning techniques for building the system. While the reported works demonstrated the effectiveness of various artificial intelligence algorithms, only a few of them have utilized the time-series information of network traffic data. Also, categorical information of network traffic data has not been included in neural network-based approaches. In this paper, we propose network intrusion detection models based on sequential information using the long short-term memory (LSTM) network and categorical information using the embedding technique. We have conducted experiments using models with UNSW-NB15, which is a comprehensive network traffic dataset. The experiment results confirm that the proposed method improves the performance, with a binary classification accuracy rate of 99.72%.
A Knowledge Completion Approach using Rule Generation based on Neuro-Symbolic Method
Jea-Seung Roh, Won-Chul Shin, Hyun-Kyu Park, Young-Tack Park
http://doi.org/10.5626/JOK.2021.48.4.425
A knowledge graph is a structured representation of real-world knowledge and is designed by collecting information from various sources. These knowledge graphs are networks that represent relationships between data and are applied in various fields of artificial intelligence; however, there exists problems related to incomplete knowledge due to the omission of entities or omission links between the entities. To resolve the problems, research on automatic knowledge completion techniques is necessitated. Consequently, various studies have been examined including embedding techniques, deep learning or symbolic rule inference using ontology. Although automatic knowledge completion can be efficiently performed through the above-mentioned methods, deep learning methods require a large amount of learning data due to data-driven processing methods, and there exist problems related to the results that are hard to explain. Futhermore, ontology-based methods require ontology and rules that are defined by the experts. To overcome this limitation, in this study, we propose an automatic knowledge completion method by explicitly extracting the implicit rules from the data based on the Neuro-Symbolic method. For rule extraction, we have implemented a symbolic unification based embedding path and defined a cost function for it to automatically generate the rules. Compared with the approaches presented in previous embedding studies, the proposed method demonstrates the superiority of the Neuro-Symbolic method concerning speed and performance. To assess the performance of the proposed method, for datasets like Nations, UMLS, and Kinship, experiments were conducted in comparison with the approach of the state-of-the-art knowledge completion studies. Consequently, an immense reduction in the training time and 37.5%p increase in the average performance were observed.
Calibration of Pre-trained Language Model for the Korean Language
Soyeong Jeong, Wonsuk Yang, ChaeHun Park, Jong C. Park
http://doi.org/10.5626/JOK.2021.48.4.434
The development of deep learning models has continuously demonstrated performance beyond humans reach in various tasks such as computer vision and natural language understanding tasks. In particular, pre-trained Transformer models have recently shown remarkable performance in natural language understanding problems such as question answering (QA) tasks and dialogue tasks. However, despite the rapid development of deep learning models such as Transformer-based models, the underlying mechanisms of action remain relatively unknown. As a method of analyzing deep learning models, calibration of models measures the extent of matching of the predicted value of the model (confidence) with the actual value (accuracy). Our study aims at interpreting pre-trained Korean language models based on calibration. In particular, we have analyzed whether pre-trained Korean language models can capture ambiguities in sentences and applied the smoothing methods to quantitatively measure such ambiguities with confidence. In addition, in terms of calibration, we have evaluated the capability of pre-trained Korean language models in identifying grammatical characteristics in the Korean language, which affect semantic changes in the Korean sentences.
A Deep Learning-based Two-Steps Pipeline Model for Korean Morphological Analysis and Part-of-Speech Tagging
http://doi.org/10.5626/JOK.2021.48.4.444
Recent studies on Korean morphological analysis using artificial neural networks have usually performed morpheme segmentation and part-of-speech tagging as the first step with the restoration of the original form of morphemes by using a dictionary as the postprocessing step. In this study, we have divided the morphological analysis into two steps: the original form of a morpheme is restored first by using the sequence-to-sequence model, and then morpheme segmentation and part-of-speech tagging are performed by using BERT. Pipelining these two steps showed comparable performance to other approaches, even without using a morpheme restoring dictionary that requires rules or compound tag processing.
Joint Sphere based 3D Animation Motion Authoring for Joint Units
Jieun Lee, Taehwan Kwon, Youngho Chai
http://doi.org/10.5626/JOK.2021.48.4.453
Research that creates natural human movement in the motion work of 3D animation is progressing in various fields with the development of technology. For more natural and realistic animation, numerous animators have worked directly on the Key-frame or received motion capture to author the motion. However, since the process of making several keys per second or measuring the motion with a sensor is an inefficient process despite simple motion authoring and modification, it is necessary to solve the problems for simple motion modification. In this study, we have analyzed the existing motion authoring method, Key-frame animation, and motion capture. Also, we propose a new motion authoring method that complements the disadvantages of the existing method. The human movement is recorded through the joint - sphere attached to each joint of 3D character and the recorded pattern and motion are revised. As the motion is modified through trajectory modification, the rotation angle and joints of each part of the model do not need to be adjusted one by one. Apparently, there is an increase in convenience along with the reduction in the working time compared to the existing motion authoring system.
Automatic Data Augmentation for Named Entity Recognition using a Text Infilling technique and Generative Adversarial Network
Cheon-Young Park, Kong Joo Lee
http://doi.org/10.5626/JOK.2021.48.4.462
Deep neural networks have been widely used in many NLP applications, However, successful construction of deep networks requires a large training corpus. Collecting a large training corpus that contains label information such as named entities is difficult and leads to a lack of data. Automatic data augmentation represents a solution to data scarcity problem. In this paper, we propose an automatic data augmentation technique for named entity recognition(NER) based on a text infilling model and generative adversarial networks. A text infilling model is used to fill missing components of a template to generate complete sentences. Using the text infilling model, we can fill in the blank of the template to generate complete and semantically coherence text with accurately named entity labels. Sentences generated by our model show lower perplexity and higher diversity than those generated in the previous approaches. Also text augmentation based on our model can improve the performance of a conventional NER system.
An Effective Detection Method of Anomalous Sequences Considering the Occurrence Order and Time Interval of the Elements
http://doi.org/10.5626/JOK.2021.48.4.469
Recently, a rapid generation of sequence data consisting of elements in various applications has been witnessed over time. Although various methods for detecting anomalous sequences among the given sequences have been actively studied, most of them mainly consider only the occurrence order of the elements. In this paper, we propose an effective anomalous sequence detection method considering not only the occurrence order of the elements but also the time interval between the elements. Apparently, the proposed method uses a model that combines two autoencoders. The first is an LSTM autoencoder, which learns the features of the occurrence order of elements, and the second is a graph autoencoder, which learns the features of the time interval between the elements. After completion of the training, each sequence is input to the trained model and reconstructed by the trained model. If the occurrence order and time interval of elements in the reconstructed sequence greatly differ from those in the original sequence, the corresponding sequence is determined as an anomalous sequence. Through various experiments using synthetic data, we confirmed that the proposed method can detect anomalous sequences more effectively than the method that uses an RNN autoencoder to learn the occurrence order of the elements, the methods that use a single LSTM autoencoder and the method that doesn’t use deep learning model.
Search
Journal of KIISE
- ISSN : 2383-630X(Print)
- ISSN : 2383-6296(Electronic)
- KCI Accredited Journal
Editorial Office
- Tel. +82-2-588-9240
- Fax. +82-2-521-1352
- E-mail. chwoo@kiise.or.kr