Recently, I co-authored an open-access journal paper that was published at Multimodal Technologies and Interaction, sponsored by the MDPI. The paper is entitled “Extended Reality Educational System with Virtual Teacher Interaction for Enhanced Learning”. The paper introduces an interactive XR intelligent assistant featuring a virtual teacher that interacts dynamically with PowerPoint presentations using OpenAI’s ChatGPT API.
It incorporates multilingual speech-to-text and text-to-speech capabilities, custom lip-syncing solutions, eye gaze, head rotation and gestures. Panoramic images can be used as a sky box giving the illusion that the AI assistant is located at another location. Findings from three pilots indicate that the proposed technology has a lot of potential to be used as an additional tool for enhancing the learning process.
More information: