CrossSense

CrossSense is an AR app for smartglasses to support individuals with Mild Cognitive Impairment (MCI) and early- to mid-stage dementia, helping them maintain independence and live fulfilling lives. By using AR to identify objects in the environment and overlay them with multisensory cues, CrossSense may help slow down memory decline and even improve broader cognition. The digital layer of, for example, name-tags, reminders, and voice reminders, reinforces memories and enables individuals living with MCI and dementia to feel grounded and carry out daily tasks with greater ease. The experience is facilitated by a friendly AI assistant, Wispy, who supports users in creating personalised associations between objects, people, and their environment. CrossSense integrates multiple types of machine learning, including object recognition, depth perception, natural language processing, and vision language models, to deliver a comprehensive and adaptive solution. We are advancing our existing proof-of-concept using openly available pre-trained models, combined with spatial understanding system (SLaM by Snapdragon Spaces), deployed to an OpenXR standard for Argo smartglasses. We partnered with DigiLens to develop on their pre-release smartglasses unit, and participate in the NVIDIA Inception Program to maximise artificial intelligence potential of our solution.