Morpheme-based Korean Word Vector Generation Considering the Subword and Part-Of-Speech Information

Junyoung Youn, Jae Sung Lee

http://doi.org/10.5626/JOK.2020.47.4.395

Word vectors enable finding the relationship between words by vector computation. They are also widely used as pre-trained data for high-level neural network programs. Various modified models from English models have been proposed for the generation of Korean word vectors, with various segmentation units such as Eojeol(word phrase), morpheme, syllable and Jaso. In this study, we propose Korean word vector generation methods that segment Eojeol into morphemes and convert them into subwords comprising either syllable or Jaso. We also propose methods using Part-Of-Speech tags provided in the pre-processing to reflect semantic and syntactic information regarding the morphemes. Intrinsic and extrinsic experiments showed that the method using morpheme segments with Jaso subwords and additional Part-Of-Speech tags showed better performance than others under the condition that the target data are normal text and not as grammatically incorrect.