In our first Partner Interview, Spyros Polychronopoulos from ADAPTIT S.A. discusses their role in developing the AR Theatre application for the VOXReality project. As XR technology experts, ADAPTIT has been deeply involved in the design and deployment process, ensuring that the technology aligns with live theatre needs. They’ve focused on user-friendly interfaces, seamless integration with theatre systems, and secure data protocols to protect intellectual property. Spyros also highlights strategies for future-proofing the application, including modular design and cross-platform development, with plans to adapt to emerging XR technologies and broaden access to theatre through affordable AR devices.
What is your specific role within the VOXReality Project?
Our organization, in our capacity as XR technology experts, has undertaken the design, development and deployment of the AR Theatre application. We have been engaged in the design process since the early beginning, e.g. in discussing, interpreting and contextualizing the user requirements. Throughout the process, we have been in close contact with the theatrical partner and use case owner, offering technological knowledge transfer to their artistic and management team. This work frame for operations has proven critical to facilitating team-based decision-making during design, and thus keeping in view the needs of both the XR technology systems and the theatrical ecosystem.
To facilitate our communication in an interdisciplinary team and to consolidate our mutual understanding, we have taken the lead in creating dedicated applications as deemed necessary.
Firstly, to render the VOX Reality capabilities in tangible, everyday terms, we created an easily distributable mobile application which demonstrates the VOX Reality models one by one in a highly controlled environment. This application can also function as a dissemination contribution for the VOX Reality project goals. We proceeded with developing a non-VOX Reality related AR application to practically showcase the XR device capabilities to the theatrical partner, and more specifically, to the team’s theatrical and art director with a focus on the device’s audiovisual capabilities.
Furthermore, we combined the two previous projects in a new AR-empowered application to better contextualize the VOX Reality services to a general audience which is unfamiliar with AR. Since that milestone, we have been developing iterations of the theatrical application itself with increasing levels of complexity. Our first iteration was an independent application running on the XR device which simulates the theatrical play and user experience. It was produced in independent mode for increased mobility and testing and was used extensively for documenting footage and experientially evaluating design alternatives. The second iteration is a client-server system which can allow multiple XR applications to operate in sync with each other. This was performed for simulated testing in near-deployment conditions during development and was targeted on evaluating the more technical aspects of the system, like performance and stability. The third and last iteration will incorporate all the physical theatrical elements, specifically the actors and the stage, and will involve the introduction of yet new technology modules with their own challenges.
In summary, this has been a creative and challenging journey so far, with tangible and verifiable indicators for our performance throughout, and with attention to reusability and multifunctionality of the developed modules to reinforce our future development tasks.
As for my personal involvement, this has been a notably auspicious coincidence, since I myself am active in theatrical productions as a music producer and devoted to investigating the juncture of music creation and AI.
What considerations went into selecting the technology stack for the theatre use case within VOXReality, and how does it align with the specific requirements of live theatrical performances?
Given the public nature of the theatrical use case, the user facing aspects of the system, specifically, the XR hardware and XR application user interface, were an important consideration.
In terms of hardware, the form factor of the AR device was treated as a critical parameter. AR glasses are still a developing product with a limited range of devices that could support our needs. We opted for the most lightweight available option with a glass-like form to achieve improved comfort and acceptability. This option had the tradeoff of being cabled to a separate computing unit, which was considered of least concern to us given the seating and static arrangement in the theatre. In more practical terms, since the application should operate with minimal disturbance in terms of head and hand movement, in silence and in low light conditions, we had decided that any input to the application should be made using a dedicated controller and not hand tracking or voice commands.
In terms of user interface design, we selected a persona with minimal or no XR familiarity and that defined our approach in two ways: 1) we chose the simplest possible user input methods on the controller and we implemented user guidance with visual cues and overlays. We added a visual highlight to the currently available button(s) at any point and in the next iteration, we will expand on this concept with a text prompt on the functionality of each button, triggered by user gaze tracking. 2) we tried to find the balance between providing user control which allows for customization/personalization and thus improved comfort, and limiting control which safeguards the application’s stability and removes cognitive strain and decision-making from the user. This was addressed by multiple design, testing and feedback iterations.
How does the technical development ensure seamless integration with existing theatre systems, such as lighting, sound, and stage management, to create a cohesive and synchronized production environment?
As in most cases of innovative merging of technologies, adaptations from both sides of the domain spectrum will need to be made for a seamless merger. One problematic area involves the spatial mapping and tracking system needs for XR technology. Current best practices for its stable operation dictate conditions that typically do not match a theatrical setup: it requires well-lit conditions, stable throughout the experience, performs best in small/medium sized areas, needs surfaces with clear and definite traits that avoid specific textures, etc. Failure of the spatial mapping and tracking system can lead to misplaced 3D content which no longer matches the scenography of the stage and thus breaks immersion and suspension of disbelief for the user. In some cases, failure may also lead to a non-detection or inaccurate detection of the XR device controller(s), thus impeding user input.
To amend this, recommendations for the stage’s scenography can be provided from the technical team to the artistic team. Examples are to avoid reflective, transparent, or uniform in color (especially avoiding the color black) surfaces, or surfaces with strong repeating patterns. Recommendations can also address non-tangible theatrical elements, like the lighting setup. Best practices advise avoiding strong lighting that produces intense shadows or dip areas in total or near-total darkness.
Furthermore, there are spatial tracking support systems that a director may choose to integrate in experimental, narrative or artistic ways. One example is the incorporation of black-and-white markers (QR, ARUCO, etc) as scenography elements which have the practical function of supporting the accuracy of the XR tracking system or extending its capabilities (e.g. tracking moving objects).
Going even further, an artistic team may even want to examine a non-typical theatre arrangement which can better match the XR technology needs and lead to innovative productions. On example is the round theatre setup, which has a smaller viewing distance between audience and actors and an inherently different approach to scenography (360° view). Other even more experimental physical setups can involve audience mobility, like standing or walking around, which can make even more use of the XR capabilities of the medium in innovative ways, like allowing the users to navigate a soundscape with invisible spatial audio sources or discover visual elements alongside pre-designed routes or from specific viewing angles.
In terms of audio input, the merger has less parameters. Currently, users are listening to the audio feed from the theatre stage’s main speakers and are receiving no audio from the XR device. Innovative XR theatre design concepts around audio could involve making narrative and artistic use of the XR device speakers. This could e.g. be an audio recording of a thought or internal monologue that, instead of being broadcasted from the main stage, plays directly on the XR device speakers, and thus very close to the viewer and in low volume. It could be an audio effect that plays in waves rippling across the audience or plays with a spatialized effect somewhere in the hall, e.g. among the audience seating. Such effects could also make use of the left-right audio channels thus giving a stronger sense of directionality to the audio.
The audio support could also be used in more practical terms. VOX Reality currently supports provision of subtitles in the user’s language of choice. In the future, we could extend this functionality to provide a voice over narration using natural-sounded synthetic speech in their language of choice. This option would better accommodate people which prefer listening over reading for any physiological or neurological reason. This feature would require supplying XR devices with noise-cancelling headphones, so that the users may receive a clear audio feed from their XR devices, isolate the theatrical stage main speakers’ audio feed and not produce audio interference to each other.
In summary, we are in the fortunate position to not only enact a functional merger of the XR technology and the art of theatre domains as we currently know them, but also to envision a redefinition of conventions that have shaped the public’s concept of theatrical experiences for centuries through the capabilities of XR. We would summarize these opening horizons in three broad directions: 1) an amplification of inclusivity by being able to provide customizable individualized access to a collectively shared experience, 2) an amplification and diversification of the audiovisual landscape in the theatrical domain and 3) an invigoration of previously niche or an invention of totally new ways for audience participation in the theatrical happenings.
Given the sensitive nature of theatrical scripts, what security protocols have been implemented to protect against unauthorized access?
Although our use case does not manage personal or sensitive medical data as in the domains of healthcare or defense, we meticulously examined the security of our system in terms of data traffic and data storage with respect to the intellectual property protection needs of the theatrical content. To cover the needs of the theatre use case, we designed a client-server system with clients operating on the XR devices of the audience and the server operating on a workstation under the assignment of the interdisciplinary facilitation team (developer team and the theatre’s technical team). As context, core reasons for the existence of the client-server system in summary were 1) to centralize the audiovisual input from the scene (microphone and video input) in order to safeguard input media quality, 2) to simultaneously distribute the output to the end-user devices in order to assure synchronicity in the audience and 3) to offset the demanding computational needs to a more powerful device in order to avoid battery and overheating issues on the XR devices.
In terms of data traffic security, the server and the clients are connected to the same local Wi-Fi network, protected by a WPA2 password, and communicate using a WebSocket protocol for frequent and fast communication. The local Wi-Fi network is for the explicit use of the AR theatre system and accesible only to the aforementioned devices, as a safeguarding measure against network bandwidth fluctuations, which could negatively affect the latency of the system and in turn the user experience during the performance, and as a security measure against data traffic inception. Furthermore, for the exact same reasons, the AI services are also operating locally in the same network and are accessed using RESTful API calls, with the added protection of a secure transport protocol (https). In summary, the entire traffic is contained in a safe and isolated environment that can only be breached by an unauthorized network access violation.
In terms of data storage, it was decided that in the release version of the application, no data logs will remain in the XR devices since safeguarding against unauthorized access of the data given the temporary provision of the devices to the public without supervision was not feasible. Any data stored will be in the server device, will hold no personalized information in any form, and will be used exclusively for technical purposes, like system monitoring and performance evaluation.
Considering the rapid evolution of technology, how is the technical development future-proofed to accommodate emerging advancements, and what strategies are in place for seamless upgrades or integrations with future technologies?
In a rapidly changing technological domain like XR and AI, planning for change is an integral part of design and development. For us, this means asking questions in two directions: 1) what the future fate of the current product can be and 2) what can the product evolve to in the future with minimal effort. Answering these questions is enabled by the fact that we, as XR developers and producers of state-of-the-art XR applications, can create informed scenarios for the foreseeable future.
One such scenario that is based on financial data and trends is the growth of the XR market, and specifically the AR sector. This is expected to diversify the device range and reduce purchase costs. In turn, this can affect us by enabling the selection of even more well-suited AR glasses for theatres, it can reduce the investment cost for adoption by theatrical establishments, and it can support the popularization of the XR theatre concept in artistic circles. At the same time, theatre-goes, in their role as individual consumers, can be expected to have increasing exposure and familiarity with this technology in general. Therefore, our evaluation for the first question is that we have good reasons to expect that our current product will have increasing potential for adoption.
On the second question, our strategy is to vigorously uphold proper application design principles with explicit focus on modular, maintainable and expandable design. Operationally, we are adopting a cross-platform development approach to be able to target devices running on different operational systems using the same code base. We are prioritizing open frameworks to ensure compatibility with devices that are compliant with industry standards, thus minimizing intensive proprietary SDK use. In terms of system architecture, by separating the AI from the XR elements, we allow for independent development and evolution for each domain in their own speed and direction. By building the connections with well-established methods that are unlikely to change, like RESTful API calls, we ensure that our product is in the best position to adapt to potentially reworking of entire modules. Furthermore, we adopt a design approach with segmented “levels of XR technology” so as to be able to easily create spin-offs targeting various XR-enabled hardware as they emerge. This does not necessarily imply more powerful devices, but also more popular ones. One current example that we investigate is to single-pick the subtitles provision feature and target affordable 2D AR glasses (also called HUD glasses or smart glasses or wearable monitors) as a means of increasing theatre accessibility.
Spyros Polychronopoulos
Researcher on digital simulation of ancient Greek instruments, and lecturer, teaching music technology and image processing.
