Seeing AI

Seeing AI, a free Microsoft app that narrates the visual world for the blind and low vision community.

Seeing AI is a free app from Microsoft that narrates the visual world for the blind and low vision community. Microsoft’s mission is to empower every person on the planet to achieve more.

That the blind and low vision community (estimated size of 253 million according to WHO) is often underserved by technology and historically has reduced educational and employment opportunities. And we want to close that gap with the help of technology.

This ongoing research project harnesses the power of Artificial Intelligence to open up the visual world and describe nearby people, text, currency, colour, and objects with spoken audio. Much like a Swiss Army knife, this app provides many tools in one by leveraging the power of on-device deep learning to open many “first time in life” scenarios.

Key features include – Real-time text reading, document structure understanding, audio-based barcode locator and product recogniser, face recognition and emotion/age/gender description, currency recognition, colour recognition, audible light detector and handwriting reader.  Additionally, this app can make even other apps accessible and inclusive by describing photos while using other apps.

Since launch in July 2017, the app has improved independence by assisting users in completing over 7 million tasks independently and has been downloaded by 200,000 users. These tasks previously required relying on sighted assistance.

Seeing AI was started as a grassroots innovation project during a company-wide hackathon. Even though the project was released in 2017, many of the team members kept working for years in their passion time since 2014 to see the dream coming to reality someday.

As a development team of 3 developers, this project also highlights what a small but dedicated team is capable of achieving. This will also set an example for other companies with resources to invest in projects which aim to make the society inclusive for everybody.

Feedback from users constantly amaze and inspires Microsoft and other developers on the impact one can make with well designed assistive technologies. User comments on Seeing AI often contain words such as ‘new confidence’, ‘new opportunities’, ‘new experiences’, ‘game changer’, ‘’independence’ and ‘empowerment’.

Microsoft have opened a ‘Seeing AI Contributor’ programme to get more researchers and developers to start contributing to the project, tacking challenges which will take 6 months, 1 year or 2 years to build. And, to to speed up the pace of innovation in the area of image descriptions, Microsoft has open sourced some of the datasets, including Microsoft COCO (Common objects in context).  This dataset took 70,000 hours of human labelling and contains 300,000 images with alt text.