Posts in category: Blog post

F6S did a great job with all the preparation and implementation of the Open Call opportunities for Third Parties. I really think that we should be particularly proud of these great results achieved by TPs as we have been also part of this process. I must say, sometimes, it wasn’t easy. We have faced several challenges and issues, but my perception was that no matter what we will find a way out and, in the end, we will showcase great results. Now, I think if you ask TPs about their experience with us there will be only humble and I believe positive feedback. And just to make it more poetic I guess TPs success is our success, as we have been there for them.

HOLO initially developed a preliminary version of an intuitive XR training application for virtual assembly tasks. This early design relied heavily on assistive cues—such as color coding, guiding lines, automatic snapping, and audio prompts—which enabled users to complete assembly procedures quickly and with minimal effort.

While this approach demonstrated the potential of XR to simplify task execution, it also revealed critical limitations for industrial training contexts. Specifically, such systems ignore the knowledge gap: assembly line workers, construction personnel, and shopfloor employees might not have previous experience with XR technology and virtual interaction. Additionally, the user may complete tasks in the virtual environment without genuinely learning or internalizing the underlying assembly logic or machinery operation. In other words, while task completion is facilitated, procedural understanding and skill transfer to real-world contexts remain limited.

Furthermore, the reliance on hand-based interactions and hand menus presents practical drawbacks in industrial scenarios. Workers’ hands are often already occupied with physical tasks, making simultaneous interaction with virtual menus cumbersome and disruptive.

To address these challenges, HOLO’s contribution within the VOXReality project, specifically in the Virtual Training Assistant use case, introduced several significant innovations:

Free-Mode Assembly Training:
Unlike the earlier stepwise, rigid training approach, HOLO implemented a free-mode assembly system. This allowed trainees to explore and attempt logical assembly sequences independently, fostering deeper understanding of the assembly process rather than mere task completion.

Voice Interaction through ASR Integration:
HOLO integrated Automatic Speech Recognition (ASR) technology developed by UM, enabling natural speech input from users. Spoken commands were transcribed into text and processed by the XR system and the AI assistant.

AI Assistant Support:
A conversational AI agent was embedded into the training workflow, providing on-demand, context-aware support based on user requests. Importantly, the agent offered assistance without unnecessary interruptions, ensuring that users retained control of the training process.

Hands-Free Interaction:
By shifting the primary mode of interaction from hand-based menus to voice-based dialogue, the usecase freed users’ hands for assembly tasks. This design improvement addressed one of the key practical barriers for XR adoption in industrial training scenarios.

Together, these innovations transformed HOLO’s training application from a task-completion aid into a more effective learning tool, capable of supporting skill acquisition, promoting logical reasoning, and better aligning XR training with real-world industrial needs.

One of the key challenges in enabling real-time multilingual communication in VR is efficiency. When many users are speaking, listening, and requesting translations at the same time, the system must manage multiple audio and text streams without overloading servers or wasting GPU resources.

In VOXReality, we solved this challenge by introducing an intermediate socket layer (middleware) between the speaker and the translation model. This middleware acts as a smart traffic controller for translations, ensuring that system resources are only used when necessary.

Here’s how it works:

• When a user activates their microphone, a dedicated speaker socket opens (/speaker/{user-id}), making their audio stream available.
• Listeners who want a translation connect through a listener socket (/listener/{user-id}/{language}), requesting subtitles in their preferred language.
• Only when both a speaker and at least one listener are connected does the middleware activate a third connection to the transcription–translation model.

This design ensures that translations are processed once per speaker and once per language, no matter how many people are listening. That means hundreds of users can receive the same subtitles in real time, without duplicating the workload.

The middleware also dynamically closes connections when they are no longer needed. If a speaker turns off their microphone, their socket closes automatically. If all listeners for a language drop out, that translation stream is shut down too. This adaptive behavior keeps the system light and efficient, saving GPU cycles for when they’re really needed.

Beyond efficiency, the middleware architecture allows the system to scale naturally. Multiple speakers can be active at the same time, each with their own user ID, translation streams, and listeners, without interfering with one another.

For presentation settings such as the Conference Room, the same architecture has been adapted for one-to-many translation. In this mode, only one person—the presenter—has the microphone at any given time. When a user steps onto the virtual stage, the system automatically designates them as the active presenter and opens the standardized endpoint /speaker/presentation. Audience members simply click the translate button on their panel to join /listener/presentation/{language} in their preferred subtitle language. The middleware then activates the transcription–translation model only when both the presenter and at least one listener are connected. Subtitles appear in real time above the stage for the audience, while the presenter sees them in front of their camera view, like movie captions.

Hello Ana Rita! As we say, CERTH-ITI, and especially the Visual Computing Lab, has a well-known pedigree in Computer Vision and graphics-related AI. The difference with VOXReality was that we had to adopt the practice of synergizing with other partners, also experts in their field such as the University of Maastricht, to a much deeper technical level than we usually do – practically on the same models and agents – to attack the multi-modal challenge we were tasked with. It was a great journey for both ends, I believe.

Hippolytus (in the Arms of Aphrodite) is an AR live performance developed for the Augmented Theatre use case of VOXReality, directed by Yolanda Markopoulou. The work experiments with AR headsets to merge stage action, visual effects, and soundscapes into a single experience where the storytelling unfolds both physically and digitally. It was produced as a 15-minute performance, engaging the artistic, dramaturgical, and technical functions of theatre rather than being treated as a technology demo.

Director Yolanda Markopoulou is an Athens-based director, producer, and creator of works that combine theatre, film, public space, and immersive media. Her projects often test how audiences move between real and imagined environments, challenging the conventional separation between performer and spectator.

Her exploration of XR began with White Dwarf (2022), a 360° VR project that placed viewers inside the Manhattan Project laboratory and showed how virtual environments can alter theatrical language. In Hippolytus (in the Arms of Aphrodite) she continued this exploration through AR, working with live and digital layers to create a unique theatre experience.

Exploring XR in Theatre – What inspires creative exploration into XR technologies in theatre?

Markopoulou explains that every XR project in theatre almost begins from zero. There are few existing models to draw on, which for her is an opportunity: “This is very positive, because you can start fresh and your imagination takes the lead.”

In traditional theatre the audience sits still, watching from one perspective. XR alters that condition: spectators remain in the same room but gain mobility, and digital environments are created around them. “They can stay in the same location, but they feel they are transported somewhere else because of the environment change.”

She describes this as working with new means of immersion: “You put the audience into different locations, into different lighting situations, sound situations, and your story develops through this kind of immersion.”

In what ways does AR differ from earlier experiments with VR, both on a creative and a technical level?

Markopoulou’s first steps into XR media were through VR using headsets, an enclosed format that placed audiences inside a virtual world. “With VR you actually close all your vision through the headset. You cannot really see the physical world,” she notes. The experience taught her the basics of presence and movement inside a virtual space, but it also highlighted how completely the headset separated the spectator from the stage.

AR altered that relationship. The Magic Leap headset is lighter and more open, keeping the physical world in view while adding digital elements on top. “You could see the physical world just with a filter, so you still felt connected to what was happening around you. It’s like you have a small screen in front of your eyes that you move around with, uncovering the world in fragments.”

AR is also unique in the way the actors themselves are able to work since they perform without being able to see the overlays themselves. “Everything had to be precisely coordinated. We relied on cues in light and sound, and with the VOXReality technologies the actors could also trigger changes in the environment around them to drive the story forward”.

What new possibilities do XR technologies open up for theatre as a medium?

For Markopoulou, XR opens new ways of working with immersion. “In XR performances the audience doesn’t just watch a story, they enter it, moving through changing environments of imagery, light and sound where the narrative unfolds.” Digital layers allow these shifts to happen inside a single performance space, giving theatre an expanded dramaturgical vocabulary.

Developing Hippolytus (in the Arms of Aphrodite) showed how AR pushes theatre into new territory, where even constraints become part of the vocabulary; a clear example of the medium as a space of exploration. “The restricted field of view encouraged audiences to move their head and body to uncover digital elements. We decided to use this feature to our advantage, turning perception itself into an active, personal process,” she notes. Sound was treated the same way: using headset audio would have been disruptive for other viewers and the actors, while headphones would restrict movement. Thus the team designed a surround system that would be free of these restrictions. “We divided the soundscape into different moments, directions, and volumes to make the experience as immersive as possible and leave the audience free to move in the space.”

Markopoulou also emphasizes accessibility as a vital aspect of XR’s potential. “I think this is a huge advantage, you don’t exclude people that don’t understand the performance’s language. With XR technology and services like those developed in VOXReality, audiences can experience theatre in multiple languages, which can make the performance much more inclusive.”

At the same time, the work on Hippolytus underlined the gap between the theatre industry and XR hardware manufacturers. “High costs and limited computing power still set boundaries, you cannot restart the performance, as you would do in a film. For XR to move further into theatre practices and unleash its full potential in the medium, technology will need to develop in closer dialogue with artists”.

What was the conceptual starting point for Hippolytus (in the Arms of Aphrodite)?

In approaching Euripides’ Hippolytus for AR, Markopoulou wanted to craft a performance that was short and experimental, while still rooted in myth. “It was a complex process where many people from different backgrounds worked together to find creative solutions for technical and artistic considerations. The headset itself set the basis for our work. Realistically we had approximately 15–20 minutes of action,” Markopoulou explains, “of course you cannot condense a full play in that time but we wanted the audience to have a full experience, not just watch an excerpt. This is how the idea of Hippolytus (in the arms of Aphrodite) was born.”

In the play, two actors, Hippolytus and Aphrodite, perform for two spectators, who witness the story at close range. “You feel very intimately involved once you enter the space and suddenly you realize that you’re in an audience of two and the action takes place right in front of you, without the usual distance between the spectator and the stage”. At times the closeness felt almost overwhelming: “People are not used to having actors playing a meter away or touching them or being close to them.”

The dramaturgy was built around this intimacy. “We started by building the action with the actors before we completed the virtual world. That way, we could see what was needed”, she recalls “Digital elements were added in rehearsal to extend or redirect what happened on stage. The overlays could conceal gestures, highlight others, or shift the audience’s attention, creating a constant dialogue between the physical and the digital, sometimes giving more space to the virtual, other times to the live action.”

In the same fashion, the development of the performance was a dialogue between artistic vision and the possibilities and limits of the technology. “We had to take many factors into account: headset performance, overheating, lighting, sound, subtitle placement, and pacing. Our goal was to maximize what was possible and give the audience as much freedom as we could, surrounding them with digital scenography that encouraged them to explore, look up, down, and around, and experience the potential of AR theatre to the fullest. It was a challenge, but we had a strong team and crucial support from the VOXReality project Maggioli Group, our technology partner, as well as from AEF.”

Master’s Student: Gabriele Princiotta
Thesis Advisors (TUM): Dr. Sandro Weber, Prof. Dr. David Plecher
Thesis Advisor (VOXReality): Leesa Joyce

A Master’s thesis titled “Exploring User Interaction Modalities for Open-Ended Learning in XR Training Scenarios” by Gabriele Princiotta was recently defended at the Technische Universität München (TUM). The thesis was co-advised by Dr. Sandro Weber and Prof. Dr. David Plecher from TUM, and Leesa Joyce from the VOXReality consortium.

This study explores how different interaction modalities affect user experience in open-ended training environments using XR. Specifically, the research focused on an AR assembly training application developed for the Microsoft HoloLens 2. Two interaction methods were designed and compared: a traditional hand-based Graphical User Interface (GUI Mode) and an AI-powered voice interaction mode (Voxy Mode), supported by a LLM and Automatic Speech Recognition.

The user study employed a within-subjects design to evaluate the impact of these modalities on user experience, cognitive load, usability, and task engagement. While quantitative findings showed significantly faster task completion times in GUI Mode—primarily due to shorter onboarding and user familiarity—no statistically significant differences emerged across other user experience metrics. This outcome was influenced by a strong learning effect throughout the study sessions.

However, qualitative feedback indicated a clear user preference for the Voxy Mode. Participants highlighted the engaging, supportive nature of interacting with the conversational AI assistant (named ARTA), noting how it made the training feel more natural and less mechanical. At the same time, the limitations of current ASR accuracy and the assistant’s understanding of nuanced or ambiguous user input were seen as key areas for future development.

AIXTRA, one of the winners of the VOXReality Open Call, has successfully completed a major testing phase. From July 21st to 25th in Leipzig, 34 participants took part in both single-user and multi-user VR training scenarios. The goal was to assess the user experience and technical performance of AI-powered VR training environments. Special attention was given to the AI trainer and voice processing features, with the team gathering end-user perceptions, subjective feedback, and objective data to ensure statistically meaningful insights.

A Diverse Group of Participants

The study’s participants, ranging from 18 to 44 years old, came from various backgrounds with differing levels of familiarity with AI and XR technologies. While many had professional experience with technology, most were “not much experienced in XR.” Concerns about AI were common, with participants frequently citing “data privacy and security, inaccuracy of AI-generated answers, dependence on AI reducing human skills, ethical considerations, bias in AI decisions, and environmental impact.” Reflecting this apprehension, a significant portion of participants (41.67%) had only “neutral trust” in AI systems.

Participants’ native languages included German, English, Ukrainian-Russian, Dutch, and Vietnamese. While English proficiency ranged from beginner to fluent, many users acknowledged their accent influenced how well they were understood. “Some noted a mixed or regional accent… responses ranged from ‘never’ to ‘sometimes’ when asked if they had difficulties being understood.”

Feedback from the testing sessions offered invaluable insights into the AIXTRA system’s real-world performance. In the single-user mode, participants found the AI assistant helpful, with one stating, “it felt intuitive and safe to work with the AI assistant” and that it “helped me manage tasks better.” However, technical challenges also emerged. A common issue was the need for users to “repeat or rephrase their inputs” to be understood. One participant experienced this firsthand, recounting, “I had difficulties pronouncing the English word ‘Barometer’ correctly, and the AI only recognised it after the 4th or 5th attempt.”

In the multi-user environment, users noted translation delays and occasional inaccuracies. One participant pointed out an issue with audio quality, stating, “Poor sound quality of the assistant was difficult”, highlighting how hardware can affect clarity and immersion. Another noted a lack of comprehensive audio cues, commenting, “There wasn’t audio feedback at every intermediate step”, which could impact the guided learning process.

Standout Feedback and Future Plans

The sessions yielded several memorable quotes that captured the dual nature of AI’s impact. One user in the single-user scenario noted the “Advantage: no fear of asking the soulless AI for advice and help without blaming myself”, but immediately followed with a perceived “Disadvantage: less independent thinking and reflection”. revealing a broader concern about “Over-Reliance and Complacency”. In the multi-user setting, a participant, despite pointing out a critical error, enthusiastically concluded, “Otherwise it was pretty cool”.

Following this successful testing phase, the AIXTRA project is now moving into its final stages. The team will use a structured user study, approved by an ethics committee, to combine user and developer feedback for deeper insights. The project will also make its demo application available as Open Access and has two more publications planned to increase public visibility. A scientific paper is also in the works to “evaluate the results and show future trends in the field of AI and XR environment”.

WELD-E, one of the winners of the VOXReality Open Call, is a project that’s pioneering the future of human-robot collaboration in the welding industry. By integrating artificial intelligence (AI) with extended reality (XR), WELD-E has created a safer, more efficient, and more intuitive welding environment. The project’s team recently presented two key papers that detail the system’s advancements.

Published Papers

The WELD-E team has published two papers outlining their work. Each paper details a different aspect of the project’s technology and its implications for the future of manufacturing.

Paper 1: “WELD-E: Enhanced-XR Human-Robot Interactive Collaboration for Welding Operations “

• Lead Authors: Andreas El Saer, George Tsakiris, Leonidas Valavanis, Aristea M. Zafeiropoulou, Konstantinos Loupos, George Argyropoulos, Petros Tsampiras
• Publication: EEITE 2025, 6th International Conference In Electronic Engineering & Information Technology
• Abstract: This paper introduces WEld-e, an end-to-end solution for human-robot collaboration in welding. It leverages AI and XR technologies, including Microsoft HoloLens, to address challenges like a lack of effective guidance and real-time monitoring. The system uses a multimodal interface with voice commands, spatial awareness, and automated decision-making. At its core, it employs deep learning models—such as Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), a domain-specific Welding Large Language Model (WeLLM), and a Visual Language Model—to execute welding commands with high precision and safety. The framework supports multilingual, context-aware interactions and improves operational efficiency and safety in hazardous industrial settings. The paper validates that WEld-e can significantly reduce setup times, training requirements, and error rates, aligning with Industry 5.0 objectives.
• Key Breakthrough: By combining AI and Extended Reality, welders can interact naturally with robots using voice and gestures. This makes welding operations faster and more intuitive by reducing errors and improving safety, all without the need for complex programming.

The VAARHeT project, one of the winners of the VOXReality Open Call and a pioneering initiative in enhancing cultural heritage experiences through technology, recently conducted successful usability testing of three innovative solutions at the Āraiši Ezerpils Archaeological Park in Cēsis, Latvia. From July 14th to 16th, 2025, visitors had the opportunity to engage with XR applications, providing feedback that will shape the future of museum interaction.

Diverse Technologies for a Richer Experience
“The study consisted of a usability test and user experience study of three AI-assisted voice-driven interactive XR applications for open-air museum visitors”, explains Cordula Hansen, a representative from XR Ireland, one of the project partners. These included:

• VR Glasses for Storytelling: These immersive glasses transport visitors back in time, narrating and visually reconstructing the captivating history of the lake castle.
• AR Glasses for Visual Translation: A different type of wearable technology, these glasses offer real-time visual translation for diverse language groups within a single tour, fostering inclusivity.
• Mobile App Avatar for Practical Information: A friendly blue-haired avatar within a mobile application provides immediate answers to common visitor queries, such as opening hours and ticket prices.

XR-CareerAssist, one of the Open Call winners of VOXReality, was presented at the International Conference on Extended Reality, Salento XR 2025, held in Otranto, Italy (June 17–20, 2025). The project uses VR and AI to make career planning dynamic, personalized, and engaging.

The paper, “Transforming Career Development Through Immersive and Data-Driven Solutions,” focuses on XR-CareerAssist to address the limitations of traditional career counseling, often seen as boring, inflexible, and hard to access.

The system uses VR goggles and AI to create an immersive environment. Users interact with a 3D avatar that understands multiple languages, visualize career paths with interactive maps, and get personalized advice from a database of over 100,000 real career profiles. It also features voice commands for natural interaction.

The paper details the system’s design, built on a vast database of CVs, and describes how AI models work together. Future plans include testing with 25-40 real users.

Real-World Impact: Sarah’s Journey with XR-CareerAssist

Consider Sarah, a 35-year-old IT manager in the UK with 10 years of experience, aiming to become a Chief Officer. XR-CareerAssist helps her directly:

• Immersive Start: Sarah puts on a MetaQuest 3 headset and enters a virtual environment with a 3D avatar.
• Voice Interaction: She simply states her current role, experience, and Chief Officer aspiration.
• Instant Analysis: The system quickly finds 1500 similar profiles, showing that such professionals typically work across 3 sectors and that reaching Chief Officer takes about 15 years.
• Visual Career Map: Sarah sees an interactive Sankey diagram (flowchart) displaying various paths from Manager to Chief Officer over the next 10 years, including industry detours.
• Multilingual Support: The system automatically translates everything into French if Sarah prefers, due to her work with a French company.
• Personalized Insights: The AI-powered avatar explains visualizations, points out necessary skills, and highlights high-success career moves. Sarah can actively ask questions via speech.

This example shows how XR-CareerAssist offers data-backed, visual, and highly personalized guidance, empowering users to make informed career decisions.