Ju Yong Sim  Seong Hwan Kim

10.5626/JOK.2024.51.1.67

Large Language Model GPT voice phishing detection prompt design

In this paper, we introduce a novel approach for voice phishing call detection, using text-davinci-003, which is a recently updated model from the generative pre-trained transformer (GPT) -3.5 language model series. To achieve this, we devised a prompt to let the language model respond with an integer ranging from 0 to 10, which indicates the likelihood that a given conversation is a voice phishing attempt. For prompt tuning, hyperparameter adjustment, and performance validation,we use a total of 105 actual Korean voice phishing transcripts and 704 transcripts from various topics of general conversations as our dataset. The proposed scheme includes a function to send voice phishing alarm during a call and a function to finally determine whether the call was a voice phishing after the call ends. Performance is evaluated in five different scenarios using different types of training and test data, demonstrating an accuracy range of 0.95 to 0.97 for the proposed technique. In particular, when tested with data from sources different from those used in training, the proposed scheme performs better than the existing bidirectional encoder representations from transformer (BERT) model-based schemes.