TY  - JOUR
T1  - Analysis of the Semantic Answer Types to Understand the Limitations of MRQA Models
AU  - Lim, Doyeon 
AU  - Roman, Haritz Puerto San 
AU  - Myaeng, Sung-Hyon 
JO  - Journal of KIISE, JOK
PY  - 2020
DA  - 2020/1/14
DO  - 10.5626/JOK.2020.47.3.298
KW  - machine reading question answering
KW  - query analysis
KW  - transformer language models
KW  - answer type
AB  - Recently, the performance of Machine Reading Question Answering (MRQA) models has surpassed humans on datasets such as SQuAD. For further advances in MRQA techniques, new datasets are being introduced. However, they are rarely based on a deep understanding of the QA capabilities of the existing models tested on the previous datasets. In this study, we analyze the SQuAD dataset quantitatively and qualitatively to demonstrate how the MRQA models answer the questions. It turns out that the current MRQA models rely heavily on the use of wh-words and Lexical Answer Types (LAT) in the questions instead of using the meanings of the entire questions and the evidence documents. Based on this analysis, we present the directions for new datasets so that they can facilitate the advancement of current QA techniques centered around the MRQA models.