Improving Accessibility using Augmented Reality — Augmenta11y
The story of how we made an app to help people with dyslexia
When we started brainstorming ideas for our degree’s thesis project, we knew that we wanted to work towards improving how people interact with technology, specifically accessibility. Through preliminary research, we narrowed our focus down to visual and cognitive impairments such as dyslexia, color blindness, and partial vision loss. Here is how we created Augmenta11y, an AR solution for helping individuals with dyslexia read real-world text better.
The Why
According to Dyslexia International, at least 1 in 10 people are affected by dyslexia, i.e. more than 700 million children and adults worldwide. Being a hidden disability, it is difficult to detect and has no cure. Moreover, manifestations of dyslexia vary from subject to subject, and each person overcomes dyslexia in a different way depending on their way of learning and on other cognitive abilities.
On the web, there are some handy products that help such individuals navigate and understand the internet better. One of them is Agastya, a plugin developed by Oswald Labs. As young researchers, we loved the idea behind the plugin and wanted to translate the same concept to the real world where text could be more accessible to people on the fly with their smartphones. We got in touch with Anand Chowdhary, CEO of Oswald Labs, and told him about our idea. He was very supportive and agreed to sponsor our research.
Research
Having agreed to work together, we sifted through and read academic journals, research papers and also other research material provided by Oswald Labs. We analysed existing products for their strengths and shortcomings. We also performed an initial user study where we spoke to people who deal with dyslexia on a daily basis and tried to understand their struggles and requirements. We came to the following conclusions:
- Sans-serif fonts such as Arial, Verdana and Comic Sans are the least confusing for people with dyslexia. There are also fonts such as OpenDyslexic that are specifically designed to combat this disability
- Text and background colour combination and contrast aided in making text more legible to the users
- Users often have a problem when moving from one line to the next (they tend to skip over lines). So we allowed them to customize the line height of the text.
Introducing Augmenta11y
Augmenta11y is an augmented reality (AR) based app which helps people with learning disabilities like dyslexia easily read signage, paper, and other material in the real world. It allows users to scan text using their smartphone, which will then be superimposed with the same text but styled to suit the preferences of the user.
The app has three features:
- Reader Mode: The user can read the text unaffected by external environmental factors by tapping on a text box and entering the Reader Mode. If required, the user can also have the text read aloud to them.
- Customizable Settings: Different dyslexic users may have different requirements for styling the text that helps them read better. To facilitate customization, we allow them to adjust properties like fonts, text to background colour contrasts and line height. We also provided a Major Object Detection mode that identifies the largest pieces of recognized text and displays only those, hence avoiding a cluttered reading experience.
- History: This tool keeps track of previously detected text for future reference.
Design and Implementation
Owing to the challenges faced by the dyslexic, we wanted the user interface to be as concise and clutter-free as possible.
We designed an on-boarding sequence to introduce the user to not only the features of the app but to also ensure that they are aware of the few hidden affordances (such as the text boxes being clickable).
The majority of our team had experience solely in web development, so we decided to create Augmenta11y using React Native. It allowed us for quick cross platform development using only JavaScript and leveraging popular open source projects.
Within the first 15 days, we had our prototype up and running. The next two months were spent in testing the product and making improvements based on the feedback.
In October 2018, Augmenta11y, along with other products by Oswald Labs, was showcased at the Dutch Design Week. This event saw a footfall of more than 335,000 visitors. We received a lot of insightful feedback from this event. One of them was the need for a Reader Mode, which was already in the pipeline, that we developed and updated in a single day, making it available for further testing for the remaining days of the event.
Testing
We tied up with health organizations from various cities in India to validate our research. We were able to test it with a group of 39 students aged 8–14 years. The students were provided with two passages appropriate for each age group. One passage had to be read with the help of Augmenta11y and the other without. The time taken by the students to read each passage, and their text styling preference recorded.
We saw a 21.2% reduction in the amount of time taken to read when using Augmenta11y. It was also observed that 85.7% of the students found the OpenDyslexic font helpful and 76.9% of them preferred to have a yellow background to the text.
What’s next for Augmenta11y?
In December 2018, we participated in the Late Break Work category of India HCI where we were able to share our research with leading academicians from all over the world. We received a positive response and some interesting ideas about the direction that our research could take.
We would like Augmenta11y to have a broader reach with the following plans:
- Translations into various languages
- Customizable Reader Mode with in-depth settings
- Improved on-boarding experience that would allow users to figure out and set their personal text styling preferences instead of using commonly accepted default settings
- Zooming feature in Reader Mode
