Using Augmented Reality and Machine Learning for Captioning in Theatrical Experiences

Authors: Olga Chatzifoti, Nick Achilleopoulos, Spiros Borotis, Moonisa Ahsan, and Eleni Oikonomou

As part of the EU-funded VOXReality project, we developed a solution for enhancing accessibility and creativity in theatres leveraging Augmented Reality (AR) and Machine Learning(ML). Our system provides the audience with real-time, customizable captions in 6 languages and with audiovisual effects through AR glasses. A live performance of Euripides’ Hippolytus in Athens, Greece, was conducted for user evaluation and resulted in decisively positive user feedback, demonstrating the potential of our system. A second evaluation is planned with future improvements of the system focusing on accuracy, quality and latency for the ML components, and learning curve for the AR components.