BERT-based Two-Stage Classification Models and Co-Attention Mechanism for Diagnosing Dementia and Schizophrenia-related Disease

Min-Kyo Jung, Seung-Hoon Na, Ko Woon Kim, Byoung-Soo Shin, Young-Chul Chung

http://doi.org/10.5626/JOK.2022.49.12.1071

Noting the recently increasing number of patients, we present deep learning methods for automatically diagnosing dementia and schizophrenia by exploring the use of the novel two-stage classification and the co-attention mechanism. First, the two-stage classification consists of two steps-the perplexity-based classification and the standard BERT-based classification. 1) the perplexity-based classification first prepares two types of BERTs, i.e., control-specific and patients-specific BERTs, pretrained from transcripts for controls and patients as the additional pretraining datasets, respectively, and then performs a simple threshold-based classification based on the difference between perplexity values of two BERTs for an input test transcript; then, for ambiguous cases where the perplexity difference only does not provide sufficient evidence for the classification, the standard BERT-based classification is performed based on a fine-tuned BERT. Second, the co-attention mechanism enriches the BERT-based representations from a doctor’s transcript and a client’s one by applying the cross-attention over them using the shared affinity matrix, and performs the classification based on the enriched co-attentive representations. Experiment results on a large-scale dataset of Korean transcripts show that the proposed two-stage classification outperforms the baseline BERT model on 4 out of 7 subtasks and the use of the co-attention mechanism achieves the best F1 score for 4 out of 8 subtasks.