Minjae Mun  Gyeore Yun  Chaeyong Park  Seungmoon Choi

10.5626/JOK.2023.50.3.228

multimodal haptics Human-Computer Interaction User eXperience Mobile system real-time audio-to-tactile conversion

This study presents a real-time multimodal audio-to-tactile conversion system for improving user experiences when users play or watch first-person shooting games with a mobile device. The system detects whether sounds from the mobile devices are appropriate to provide haptic feedback in real-time and provides vibrotactile feedback, mainly used for conventional haptic feedback, and impact effects of short and strong force as well. To this end, we confirmed the suitability of the impact haptic feedback compared to the vibrotactile feedback for shooting games. We implemented two types of impulsive sound detectors using psychoacoustic features and a support vector machine. We found that our detectors outperformed the one from a previous study. Lastly, we conduct a user study to evaluate our system. Results showed that our system could significantly improve user experiences.