Voice-driven interaction in XR spaces
Challenge:
Integration challenge
AIXTRA, a consortium consisting of LEFX GmbH and MindPort GmbH, successfully addressed two significant and recurring challenges in VR-based training: overcoming language barriers and enabling rich human-to-human interactions without the need for multiple participants to be physically present. MindPort is the company behind VR Builder, the free and open source Unity plug-in for VR app development. With over 10.000 downloads and a consistent 5 star rating for almost 4 years since publishing, it is one of the most successful add-ons for VR in Unity. LEFX specializes in interactive digital formats. Since 2011, they have delivered high-quality real-time visualizations, product presentations, and VR/AR applications across several sectors. LEFX excels in enterprise-level XR and real-time 3D applications. LEFX has been a strong project contributor to VR Builder since its inception.The primary objective was to create a more effective and accessible VR training environment, ultimately advancing VR training methodologies and technologies within the broader VOXReality project. By integrating AI-based in-VR translation and an AI-based virtual training partner, AIXTRA achieved enhanced communication, make learning more inclusive, and simulate human-to-human interactions for more impactful single-user training scenarios.
AIXTRA’s core innovation is the integration of VOXReality’s AI capabilities directly into VR Builder, an open-source Unity plug-in. The system supports flexible deployment, either locally hosted or run on a private cloud backend, ensuring high data security. The entire solution is delivered as a plug-and-play Unity package, simplifying integration into existing VR Builder projects.
The conducted user study demonstrates that the developed AI capabilities were recognized as extremely helpful by VR users. In the single user scenario, the AI-based training partner was widely perceived as helpful and intuitive, enabling all single-user participants to complete their training sessions effectively, while in multi user, the real-time translation agent proved overwhelmingly positive in multi-user settings.
AIXTRA transforms VR training by integrating VOXReality technologies with the VR Builder toolset, making advanced, AI-enhanced training scenarios easier to develop and implement. Developer interviews highlighted a 3x – 4x acceleration in AI feature development, enabling a faster time-to-market for AI-enhanced products, coupled with the cost-saving and flexibility benefits of its open-source, modular, and self-hostable design. This will broaden the reach of VOXReality technologies, connecting global teams and driving the adoption of cutting-edge AI solutions.
By enabling simpler yet comprehensive training scenarios, AIXTRA enhances collaboration, inclusivity, and the overall effectiveness of VR training, contributing significantly to the future of professional development and education. This has already led to the technology’s adoption in two other large projects which instantly benefited from the value that the AIXTRA technology provides.
The advent of real-time translation, VR/XR training, multinational group VR/XR training and AI-based virtual training partners is transforming various fields by dismantling barriers and enhancing the learning experience. The implementation of real-time translation and AI-based coaches have the potential to facilitate global collaboration by enabling seamless communication between multinational teams. Furthermore, it can improve coach services by allowing agents to assist trainees in their native language and expand educational opportunities by allowing them to access content and participate in courses in different languages.
However, VR/XR training provides immersive training experiences for remote workers, simulates hazardous environments for safe, hands-on training in industries such as construction and manufacturing, and allows medical professionals to practice procedures in a risk-free virtual environment together. Artificial intelligence-based virtual training coaches facilitate personalized learning experiences by adapting training applications to individual learning styles and paces. They offer continuous training opportunities without the need for human trainers to be present and monitor and analyze performance data to identify areas for improvement and optimize training applications.
In addition, the possibility to create one training application in a specific language and auto-translate it to various languages improves the scalability of such training applications. The usage of AI-based virtual training partners to provide greater variety would also be an effective method of enhancing the experiential aspect of VR/XR training. Through our authoring tool, companies can create their own training using AI-based virtual training partners, automated real-time translation and existing functionalities of VR Builder to provide better VR/XR training experiences.
So, multi-User-Training scenario with strong focus on communication and collaboration to showcase and evaluate the realtime translation functionalities with a close-to-real-world process. As well as, single-User-Training on a similar real-world scenario, which usually would require collaboration with another human operator, which will instead be achieved by utilizing the AI-enabled functionalities of the AIXTRA-project.
Mindport
Founder & CEO
Mindport
Software Engineer
Mindport
Senior Developer
LEFX GmbH
XR-Developer
LEFX GmbH
Co-Founder & Co-CEO
LEFX GmbH
Unity Developer
Mindport
“As project manager at MindPort, the most valuable part of participating was undoubtedly seeing how we could directly tackle persistent, real-world challenges in enterprise VR training with innovative AI concepts. By collaborating closely with LEFX and the VOXReality team, we integrated components like real-time translation and an AI-based virtual training partner into our VR Builder toolset and enhanced our team’s capabilities in providing groundbreaking new AI features with a direct impact on VR training creation.
During the project we have not only strengthened the VR Builder ecosystem, giving us a unique selling point in the enterprise sector, but our work has also already been adopted in other significant projects and several clients demonstrate clear interest in adopting AIXTRA in their upcoming projects.
The feedback from our user studies and developer interviews has been crucial, providing us with positive and valuable feedback. It also reinforced the value of our open-source, self-hosted approach for building trust and ensuring flexibility, especially for high-security industries.
Personally, this project highlights the resilience and adaptability required in cutting-edge development and shows where AI can provide clear improvements in XR projects.”
Gruppo Maggioli