Real-time Multimodal Audio-to-Tactile Conversion System for Playing or Watching Mobile Shooting Games

Minjae Mun, Gyeore Yun, Chaeyong Park, Seungmoon Choi

http://doi.org/10.5626/JOK.2023.50.3.228

This study presents a real-time multimodal audio-to-tactile conversion system for improving user experiences when users play or watch first-person shooting games with a mobile device. The system detects whether sounds from the mobile devices are appropriate to provide haptic feedback in real-time and provides vibrotactile feedback, mainly used for conventional haptic feedback, and impact effects of short and strong force as well. To this end, we confirmed the suitability of the impact haptic feedback compared to the vibrotactile feedback for shooting games. We implemented two types of impulsive sound detectors using psychoacoustic features and a support vector machine. We found that our detectors outperformed the one from a previous study. Lastly, we conduct a user study to evaluate our system. Results showed that our system could significantly improve user experiences.