MTI 2024 Article

Recently, I co-authored an open-access journal paper that was published at Multimodal Technologies and Interaction, sponsored by the MDPI. The paper is entitled “Extended Reality Educational System with Virtual Teacher Interaction for Enhanced Learning”. The paper introduces an interactive XR intelligent assistant featuring a virtual teacher that interacts dynamically with PowerPoint presentations using OpenAI’s ChatGPT API.

It incorporates multilingual speech-to-text and text-to-speech capabilities, custom lip-syncing solutions, eye gaze, head rotation and gestures. Panoramic images can be used as a sky box giving the illusion that the AI assistant is located at another location. Findings from three pilots indicate that the proposed technology has a lot of potential to be used as an additional tool for enhancing the learning process.

More information:

https://www.mdpi.com/2414-4088/8/9/83

IEEE CG&A Article

Recently, a co-authored article entitled ‘XR4ED: An Extended Reality Platform for Education’ was published at IEEE Computer Graphics and Applications. It presents the overall architecture and objectives of the EU-funded project dedicated to XR for education, called Extended Reality for Education (XR4ED). The goal is to provide a platform where educators will be able to build XR teaching experiences without the need to have programming or 3D modeling expertise.

The article describes the platform and focuses on a key aspect of collaborative and social XR, which is the use of avatars. We show initial results on a) a marketplace which is used for populating educational content into XR environments, b) an intelligent augmented reality assistant that communicates between nonplayer characters and learners, and c) self-avatars providing nonverbal communication in collaborative VR.

More information:

https://ieeexplore.ieee.org/abstract/document/10640174

ERCIM News XR4ED Article 2024

Recently, a co-authored article entitled ‘Extended Reality Intelligent Assistant for Education’ was published at ERCIM News. The article introduces an intelligent Extended Reality (XR) teaching assistant communicating with the users in natural language. The application uses artificial intelligence (ChatGPT) to understand and communicate with the students using realistic-looking avatars with body language. Teachers can customise the application by adding their slides to be taught or choosing a teaching topic that will be discussed by two intelligent agents.

Although the application is primarily presented in an AR environment, it offers a panoramic image mode that can optionally be used to surround the user in the XR space. This shift towards VR along the MR spectrum brings more immersion and is especially suitable for certain learning scenarios, e.g. featuring learning about historical sites, natural ecosystems, or other content that is bound to a place. The XR Intelligent Assistant has been implemented on Magic Leap 2 AR head-mounted display (HMD) and for Android for usage on phones.

More information:

https://ercim-news.ercim.eu/en137/special/extended-reality-intelligent-assistant-for-education