Posts in category: Blog post

At its core, the project had two main goals: advancing natural language processing literature and making sure these technologies actually work inside real XR applications. To do that, not only did we work on the fundamental research but also, we developed applications to try them in real world cases. We published 16 scientific papers where we introduced new methods and models. We also made our work available to the public by releasing 25 models and 6 datasets shaped by our scientific outcomes and the needs of XR use cases. On top of that, we developed three optimization tools to make these models lighter and efficient enough to run in XR environments. From the end-users’ perspective, what really matters is how this work performs outside the lab. We demonstrated the capabilities of our models through three applications developed within the project and five more built by our external partners, each combining our models in different ways. These eight applications were tested with real end users in pilot studies, and the results gave us solid scientific evidence that the project’s achievements are not just academically sound, but also practical and valuable to be used in real-world XR scenarios. 

Extended Reality (XR) is rapidly evolving from visually immersive environments toward intelligent, responsive systems capable of adapting to human intent. A key milestone in this evolution has been achieved through the VoxReality project, which demonstrated how Artificial Intelligence (AI) can be effectively integrated into XR to enable more natural, voice-driven, and context-aware interactions. Building on these outcomes, future XR applications will offer more personalized and immersive experiences through understanding the user needs and preferences.

Looking ahead, the integration of AI into XR applications is expected to increasingly rely on advanced human-AI collaborative frameworks, enabling the emergence of XR hybrid intelligent systems in which human expertise and AI capabilities are tightly coupled within immersive environments. Immersive spaces allow AI systems to observe user actions, gestures, speech, and attention in context, while users can directly perceive and interact with AI behavior. XR hybrid intelligent systems will facilitate not only intelligent interactions and immersive experiences, but also personalised content generation that takes into account users’ knowledge, skills, and performance. For example, a virtual training assistant could dynamically generate immersive scenarios that specifically target and bridge individual knowledge gaps. Beyond content personalisation, XR hybrid intelligent systems will leverage user feedback to continuously adapt and improve system performance in alignment with evolving user needs. Key feature in these systems is to empower users in guiding, correcting, and shaping AI behavior over time.

In the context of XR hybrid intelligence, Explainable AI (XAI) and human-in-the-loop technologies become critical enablers. As XR systems grow more autonomous and intelligent, users should be able to understand why an AI agent behaves in a certain way. XAI techniques can make AI reasoning transparent within XR environments, for instance by visualising decision pathways, highlighting relevant contextual cues, or providing natural language explanations. Moreover, the integration of XAI into XR environments introduces new opportunities for sense-making and reflection. By embedding explanations directly into immersive experiences, users can not only observe system outcomes but also explore underlying reasoning processes in a spatial and interactive manner. This can transform XR from a passive visualization medium into an active cognitive workspace where users learn with AI, rather than merely from it. In parallel, human-in-the-loop approaches encourage user feedback, enabling continuous system adaptation, performance improvement, and trust calibration. Such transparency not only builds trust but also supports effective human-AI collaboration.

In conclusion, the VoxReality project represents an important step toward intelligent XR environments grounded in natural interaction. The future of XR will build upon these foundations by embracing hybrid intelligence, transforming XR into a space where humans and AI systems work together seamlessly, transparently, and creatively. This convergence will redefine how we interact with digital worlds: not as users of technology, but as partners within intelligent immersive environments.

Presenting to a global audience in VR isn’t just about visuals and sound—it’s about making sure everyone, regardless of language or location, can follow and engage. In large, one-to-many settings like keynotes or panel discussions, translation needs to be seamless, scalable, and non-intrusive.

In VOXReality, we tackled this challenge by designing a dedicated VR Conference Room: a space built from the ground up to support real-time multilingual presentations, immersive slide sharing, and interactive Q&A, all within an acoustically optimized 3D environment.

Unlike physical spaces, where sound depends on proximity, the conference room was designed with equal audio clarity across all seats. No matter where a user sits—even in the back row—they can hear the presenter just as clearly as those at the front.

To ensure a smooth presentation flow, the speaker can share any window from their device, not just a file or a tab. This live window feed is then projected onto the blackboard surface of the auditorium—not a virtual screen—enhancing immersion and realism while maintaining focus.

Here’s how the system works:

• When a speaker enters the stage, the system recognizes them as the presenter, activating automatic transcription and translation of their speech.
• Subtitles appear clearly above the blackboard in each participant’s preferred language—up to six supported—ensuring everyone receives the message in real time.
• Users can toggle subtitles on or off based on personal preference.

The conference room layout mimics a real-world auditorium, promoting attentiveness and focus. During the Q&A session, participants raise their hand virtually. The presenter then grants speaking permission to one participant at a time. As the audience member speaks, their voice is transcribed, translated, and shown in a movie-style subtitle panel directly in front of the presenter—making multilingual dialogue feel effortless and intuitive.

The objectives of the VOXReality project were to develop an advanced XR ecosystem integrating linguistic AI, computer vision, and multimodal interaction to improve multilingual immersive experiences. The goals of the project were mainly related to three spheres. One was the creation of pre-trained linguistic and visual models for XR environments. The second sphere was the development of spatiotemporal digital assistants and real-time contextual translation. The last one was validation through the three use cases: multilingual immersive theater, virtual conferences, and industrial AR training.  

The main Key project outcomes were: 

• AI-vision language models for virtual and real environments; 
• Instruments for the visualisation of 3D in Augmented Reality; 
• Multilingual speech translation models 
• XR experience for both the theatre and the conferences.

The consortium has gathered partners in different fields such as academics, industrials, and creatives, in order to combine scientific, technical, and artistic expertise. The cooperation was based on shared management of the IPR and a common vision of collaborative exploitation and open innovation.  

Throughout VOXReality, one of our most important achievements was creating virtual agents capable of operating seamlessly in real‑time, human‑centred XR environments.

A major breakthrough was our ability to integrate these capabilities directly inside live XR scenes, so agents could perceive what users were asking, respond to their intent, and adapt their answers accordingly.

AR theatre is a new and rather unexplored or undocumented domain, and there was little previous experience in the team on implementing such a performance. Therefore, we had to develop our own way of working with each other. One lesson that we face repeatedly in this regard and whose importance cannot be overstated, is that a shared language to coordinate across the team is crucial to clearly express expectations and requirements. That language may be not just textual, but also visual – it can require low fidelity mockups, role-playing, or interactive prototypes.  Devoting time for this team-building process early in the project is a worthy investment. We also learned that it was important to understand more deeply the existing workflows of theatre and try to incorporate the XR component as smoothly as possible in those pipelines, instead of suggesting an altogether new pipeline that was very robust but more inspired by software engineering rather than e.g. performative arts. For example, some tasks in the AR theatre pipeline had overlaps with tasks typically performed with commercial professional applications, and users expected commonalities or even interoperability. We would benefit from higher user engagement if we had prioritized such considerations.   

F6S did a great job with all the preparation and implementation of the Open Call opportunities for Third Parties. I really think that we should be particularly proud of these great results achieved by TPs as we have been also part of this process. I must say, sometimes, it wasn’t easy. We have faced several challenges and issues, but my perception was that no matter what we will find a way out and, in the end, we will showcase great results. Now, I think if you ask TPs about their experience with us there will be only humble and I believe positive feedback. And just to make it more poetic I guess TPs success is our success, as we have been there for them.