At the Microsoft Imagine Cup Worldwide Finals 2012, held in Sydney in July, Team Mobile Eye of New Zealand presented an application that helps people who are visually impaired become more familiar with their surroundings. The prototype Mobile Eye app —which uses both artificial intelligence (AI) and crowd-sourcing technology that enable blind people to take photos and hear audio descriptions of the resulting images — greatly impressed the panel of international judges and helped the three Auckland University of Technology (AUT) students secure a top-six finish in the global competition.
Using the Mobile Eye app, a blind person takes a photo with their smartphone, which is routed through a server and subsequently recognised using either AI or human intelligence. The colour of an object, for example, can be identified by AI, while text can be converted into audio and played back to the user via optical recognition technology.
The app also leverages the power of social media platforms, namely Facebook: a photo can be sent out to the user's community, where a friend or family member can send back an answer explaining what's in the image, which is then described out loud to the user. Through social networking, any status update can be posted immediately to a helper's profile — this flexible, crowd-sourced element of the Mobile Eye app caught the attention of one of the Imagine Cup judges, who praised the team for not attempting to solve the entire problem with technology. And as a final fallback option for users, the Mobile Eye app incorporates a third-party commercial intelligent recognition service called IQ Engines.
Team Mobile Eye's Aakash Polra, Jade Tan and In-Hwan Kim said it was Neil Jarvis of the Royal New Zealand Foundation of the Blind who inspired the concept, and the team worked closely with the Foundation to further develop their app.
The team hopes to continue developing the project with the aim of exporting the technology to help blind and partially sighted people around the globe — the architecture is globally scalable for a market of approximately 40 million blind people and 315 million who are visually impaired.
“We are so proud of Team Mobile Eye, who headed into this tough technology competition with an ingenious idea and managed to amaze the judges,” said Scott Wylie, Director of the Developer and Platform Group for Microsoft New Zealand. “They were fantastic representatives for the country, and have helped earmark New Zealand as one of the most innovative countries in the world by developing creative, cutting-edge solutions that receive recognition on the world stage.”
“Team Mobile Eye members have helped earmark New Zealand as one of the most innovative countries in the world by developing cutting-edge solutions that receive recognition on the world stage.”
- Scott Wylie, Director of the Developer and Platform Group, Microsoft New Zealand