Tools for Success: Tech Convergence and Co-Designed Products Close Gaps for Children Who are Blind

On Tech and Vision Podcast with Dr. Cal Roberts

People who are blind or visually impaired know all too well the challenges of living in a sighted world. But today, the capabilities of computer vision and other tech are converging with the needs of people who are blind and low vision and may help level the playing field for young people with all different sensory abilities. These tools can pave the way for children’s active participation and collaboration in school, in social situations, and eventually, in the workplace, facilitating the important contributions they will make to our world in their adult lives. Greg Stilson, the head of Global Innovation at American Printing House for the Blind (APH) together with partner organizations Dot Inc. and Humanware, is on the verge of being able to deliver the “Holy Braille” of braille readers, a dynamic tactile device that delivers both Braille and tactile graphics in an instant, poised to fill a much-needed gap in the Braille textbook market. Dr. Cecily Morrison, principal researcher at Microsoft Research in Cambridge UK and her team developed PeopleLens, smart glasses worn on the forehead that can identify the person whom the user is facing, giving the user a spatial map in their mind of where classmates (as one example) are in space. PeopleLens helps children who are blind overcome social inhibitions and engage with classmates and peers, a skill that will be crucial to their development, and in their lives, as they move into the cooperative workspaces of the future. 

Podcast Transcription

Akselrud: Typically, the first gross motor skill to develop in babies is lifting their head up. We smile at them and that motivates them to lift their head up. But if they’re not seeing anyone, if they’re having this visual impairment, it really impacts them from early on from that first developmental milestone.

Roberts: Dr. Robin Akselrud is a licensed occupational therapist and assistant professor at Long Island University in Brooklyn. 

Akselrud: Initially, it may be lifting their head off the mat, but as they get older, it will be interacting with other peers, taking turns for knowing their distance, right, how close they are or how far they are from someone else. They may get into someone’s personal space, which may be uncomfortable. And that may inhibit their social skills or their ability to make friends. That would affect their ability to play with others.  They may get lost in navigating their school environment, and they can become easily frustrated. 

Roberts: Dr. Akselrud says that children who are blind may develop hesitancy or inhibitions to new sensory experiences as well.

Akselrud: So as occupational therapists, we provide those children with even more sensory experiences, those learning experiences that they may be hesitant, afraid, or unable to get on their own. We also work with the child on developing social skills, through modeling, through working with other children in their classroom as peers, taking turns, teaching them those social skills that other children may just get naturally.

Roberts: Early exposure and interactions are critical for helping children who are blind or visually impaired develop social skills. But many times these children don’t have the opportunity to experience the same group activities as children without disabilities. That’s why disability consultant Bryce Weiler started The Beautiful Lives Project. 

Weiler: The Beautiful Lives Project does programs for people with disabilities across the country as a whole.  One we did with Michigan State football at the start of April, where 82 people with disabilities were able to play on the field with Michigan State’s football team and learn football skills such as throwing the football, catching the football, kicking the football.  And for the parents, just being able to see their child excel in something that the parents did not believe they could do, or even think they may have the opportunity to try it. 

I was born four months premature and developed retinopathy of prematurity. I would consider myself as being blind since I cannot see very much, except for some light and some shadows.

Roberts: Bryce fell in love with sports at a young age. 

Weiler:  I grew up listening to NASCAR on the radio, riding around on a Big Wheel and a tricycle on the driveway, and it brought to me visual pictures in my head of the cars going around on the track. As I got a little older, I started listening to Brian Barnhart, commentating Fighting Illini football and basketball.

And then Don Fisher, commentating Indiana Hoosiers football and basketball. And those two broadcasters created for me visual pictures of the game taking place on the court or the field. When I was at the University of Evansville, I learned how to analyze games on the radio. Tom Benson taught me how to do that by studying the style of the play-by-play broadcaster, who I’m commentating with, and then by memorizing facts on the players and coaches.

If I’m commentating a basketball game, I will go out on the court and shoot free throws beforehand to figure out if the rim is loose or tight and how the ball will bounce. And also make sure that there’s a basket microphone on the rim. So I can hear the ball hitting the rim or swishing through the net.

Roberts: And he’s done a similar thing when he commentated baseball and softball games as well. Putting a microphone on home plate to hear the umpire’s call. 

Weiler: So I figured out how I could email the Oriole staff in 2016 and I emailed them and explained to them that I would like to help make baseball stadiums and the experience for fans more accessible for people of all disabilities.

They gave me the opportunity to be their disability consultant, which I’ve done for the past six years now. 

Roberts: Today, Bryce supports a number of companies as a disability consultant, but his passion is the Beautiful Lives project. 

Weiler: For children, just giving them that foundation of making friendships as they’re growing up and the opportunity to be a part of something. Sports can allow them to do that. And it also gives them the chance to do things that their peers are taking part in. 

Roberts: Welcome to On Tech and Vision. I’m Dr. Cal Roberts. Today’s big idea is how new, inclusively designed technologies are changing the lives of young people who are blind. No longer will they have to deal with some of the basic life challenges that people like Bryce have had to overcome.

Plus, these innovations will also position them for academic, vocational, and social success. And, we learn how some of the brightest minds at the biggest tech companies in the world are spearheading the technology projects that can make that happen. We speak with Greg Stilson of American Printing House for the Blind, who, with his team of partners, is on the cusp of developing a dynamic tactile display for texts and graphics that would revolutionize the way that students who are blind access textbooks and possibly the web. And, Dr. Cecily Morrison, who along with her research team at Microsoft have developed PeopleLens, a tool that helps young people who are blind or visually impaired build a strong foundation in socialization skills to engage and collaborate with their peers more confidently.

Stilson: I love working with teachers. I love working with students and I believe that the best way that we can empower blind people is to give them the tools at as young of an age as possible to give them access to information. 

Roberts: Greg Stilson is head of global innovation at American Printing House for the Blind.

Stilson: Which is basically like a little startup inside of a big organization.

For those listeners who aren’t familiar, we are one of the larger nonprofit organizations. We are federally funded. There’s a federal appropriation that is based off of the census for students who are classified as blind or low vision. And there’s a set amount of money that every student who is classified as that gets from the federal appropriation, and that money iis divvied up amongst the states, and can be spent on products from the American Printing House for the Blind. And we create products from the lowest of the low tech (braille paper or slates) to really high-tech products like refreshable braille displays. And this is done through partnerships.

We also partner with folks like Vispero and Microsoft, Google, and Apple. And I would say one of the biggest accomplishments is really expanding our partnerships. We now partner with organizations and companies all over the world and really sort of changed the way that APH now develops technology.

Roberts: In our podcast episode, “The Latest Frontier in Tactile Technologies”, we covered the need for a dynamic, tactile display that will enable both tactile graphics and braille. Greg and his team are very close to delivering such a device.  

Stilson: This was what the field regards as the “holy braille,” right? Being able to have both on the same surface. And one of APH’s biggest mandates is providing textbooks. 

Roberts: And Greg brings his personal experience to this effort and every product he develops with APH, since he is also blind.

Stilson: When you think of a braille textbook, I remember my calculus books being 40 volumes and arriving two months late to my university in college. Right? So, this is unfortunately not an uncommon issue for blind folks all over the world. Why can’t we bring textbooks into the 21st century? Many students are reading textbooks on e-readers now. Why can’t a blind person do this? And we have single line refreshable braille displays today that can read text content. But a single line braille display can’t read a chart, and they struggle at reading tables, and they struggle at reading graphics.

You’ve got to remember a blind person sees things part to whole, so it takes us a long time to build that entire image in our head. We have to piece all these pieces together as we’re running our fingers across the graphic. So having additional contextual information was crucial and that’s really where braille was essential. 

Roberts: APH has partnered with Dot Inc, which has developed a pin technology that doesn’t use the Piezoelectric effect, so it can be produced less expensively, and with HumanWare to develop the software for electronic braille ready formats.

Stilson: Piezoelectric cells are significantly more expensive than the technology that we have today. And make no mistake, we’re still talking potentially, you know, $8,000 to $10,000 devices. Whereas the devices before would cost $50,000 to $80,000. The differences here are that APH has the ability, using the federal appropriation, along with seeking additional federal funding, to allow for these devices to be greatly reduced in cost. 

Roberts: So where is the device in development?

Stilson: We have a non-functional prototype that we did initial user experience testing to make sure that we put the buttons in the right place, that the weight, the footprint is all good. Everybody is happy with that. There’s a couple of tweaks that we’re making, but besides that, the general consensus is that the physical device, people are very happy with. The device is going to be about the size of a 15-inch gaming laptop. It’s a little less than five pounds. So, the next step is producing alpha units, which we’re hoping to have later this summer.

And what I’m preparing for is what my wife has grudgingly called “Greg’s world tour” which is, we’re going to set up regional user experience sessions where we’re going to go around the country and basically put this in the hands of students and professionals as well. And then one of the requirements here at APH is we have to do an official field test, and that’s six weeks of sending out a certain number of units to schools around the country, and they use it in their everyday life in the classroom for six weeks or so. So our hope is to have field testing units by October of next year, which is a really, really aggressive goal. 

Roberts: So, in many ways you have been a student twice. Once, growing up yourself and learning. Now, as a parent of sighted children.  What have you learned from watching your children learn that helps you now think differently about how children who are blind should learn? 

Stilson: That is a really good question Dr. Cal. I have a five-year-old daughter and even my one-year-old son, you know, parents today hand their kids iPads and the kid, you know, with very minimal instruction from the parent, because the user experience on these devices are so good that they can figure out how to swipe over to their home screen that has their game on it. And when they’re in the game, they just figured it out. I don’t believe that blind students are given that capability to just try out technology and create experiences on these technologies at such young ages.

Roberts: Young children learn in multiple ways, as you said, some of them independently, some of them together with their friends and their peers and their fellow students. How does your five-year-old learn from his friends and peers differently than you learned when you were five years old? 

Stilson: There’s so much done with visual replication, right? So, she sees how her friend does a cartwheel and she’s going to duplicate that because she visually sees how it’s done. I was playing T-ball in the backyard with her, and I was showing her how to swing a baseball bat. And I learned how to swing a baseball bat by my dad doing hand over hand on a baseball bat. Here’s how you hold the baseball bat. Here’s how you swing it. But I realized that all I had to do is show her. And she walked up and just did it, right? And then I did hand over hand just to make sure that she was doing it the correct way.

But I forget that sighted kids can do so much from visual replication, right? And when I talked about the iPad experience, that’s probably how they learned how to engage with this, is by watching my wife and I engage in our phones. It’s, you know, swiping gestures, multi-finger gestures, that type of stuff. 

Roberts: So, let’s take a big picture view. What would be your dream for the future of technology for people who are visually impaired?

Stilson: I think the next big frontier for blind people is this concept of augmented reality, virtual reality. And there’s so many use cases for a pair of intelligent wearable glasses for a blind person, including the gaming space. And I think we’re just touching the surface of inclusive video games. Gaming is such a big piece of the puzzle for learning with today’s sighted population. And we are starting to see an inclusion of video games for people who are blind or low vision and have other impairments as well. But I think that, you know, with all these combinations, right, with the advancements of virtual reality, AR, VR, all that kind of stuff, along with the idea of making experiences and video games and things like that more inclusive, it’s going to create a much more inclusive way for blind kids to engage with their sighted peers.

Roberts: And that’s where our next guest comes in. Dr. Cecily Morrison is principal research manager in the Future of Work community at Microsoft Research in Cambridge, UK. She didn’t begin her career as a technologist though, but as an anthropologist. Please tell us how an anthropologist is now designing artificial intelligence agents for people who are visually impaired.

Morrison: Anthropology is the studies of systems, it’s the, it’s the systems of people, the cultural systems, the systems of interaction. So, when we interact with a piece of technology and certainly nowadays, there’s almost always a piece of technology within that interaction system we have with other people. So, I was studying the core of those systems and it was only a little bit later when I had gone off to Hungary on a Fulbright scholarship, and through a number of serendipitous events, I was teaching in a school in the early 2000s using fairly, what we would now consider, early technology and realizing it just didn’t fit in the cultural system of a school. I couldn’t make it work between my students and that opened my journey into technology. And I went off and did a PhD in Cambridge.

Now, how did I get into working with people who are blind and low vision? Well, I was working in the healthcare domain. I was working a lot with computer vision technologies in the imaging space. And some of my colleagues said, well, we’d like to do something for people who are blind or low vision. And I said, well, why? And they said, well, we can solve all these really interesting problems. And I said, well, that’s not really for people who are blind or low vision, that’s for research. So why do you want to work with people who are blind or low vision? And, it was through these conversations, and I, at the time had a very young child who had been born blind that I started to bring the two things together, what people really needed with where the technology could go, and try and see in a research context: could we make those two things meet? And I think we have. 

Roberts: So, at Microsoft, you lead a group called Project Tokyo. Explain Project Tokyo for us.

Morrison: Sure. So, Project Tokyo is about creating computer vision technologies that help people who are blind and low vision understand who is in their immediate vicinity.

Roberts: And so why is that important? 

Morrison: So, when we started Project Tokyo, about five years ago, there was a real interest in what computer vision could enable. We had just the beginning of Siri, we had Cortana for Microsoft. But things like Google Look, those things didn’t yet exist. We also had a lot of research going on in the space of how AI systems could beat people. So, Google had made a big announcement about beating the Go champion. And that was the context when we started to think about, how can these technologies not compete against people, but really enable them? And what does it mean to enable people?

And that took us on a journey to think about how technologies can empower people to extend their own capabilities. And our very first exploration of that was in 2016. We followed a number of Paralympic athletes and spectators off to Rio to say, these are people with incredible capabilities. So how are they using technology to extend those capabilities? So, we did a lot of ethnographic and observational research.

We then extended that to different parts of the world from India to the UK, to the US, to Brazil. And we asked the question: what is it? And we co-designed with people. What is it that these technologies could enable? And across those very different environments with people in very different situations, we found that “people” was the thing that was most interesting to people.

And that doesn’t surprise us. We are people and we like other people. Knowing what’s around you is useful, but many people had other techniques to solve this, but it was much harder to go up and ask a person what you wanted to know about them. So, in a social context, these tools could open up a number of opportunities. They can open up asking things or understanding things that other people could see that you wouldn’t necessarily feel comfortable asking, but they also opened up opportunities to engage with other people to be part of the community. And finally, they were a good way of getting help. Why have always the technology do everything? And we found, more usefully, was to enable the technology to provide common ground between different people with different sensory capabilities. 

Roberts: So, every primate, when that primate hears a sound, or in this case, hears a voice, turns their head towards that sound. Why? Why do we turn our head towards that sound? 

Morrison: In particular, our attention and our memory are often organized spatially. And that’s true for people, whether you see or whether you don’t see. In particular, children who are born blind can have real difficulty connecting their representations in their mind with the spatial elements of the world. And that can lead to a whole range of things, with difficulties of attention being one of them. And not in the ways that we might typically see it, but difficulties, for example, in maintaining conversation, difficulties in engaging other people in conversation. In this particular case, we’re using something called the PeopleLens technology.

It’s a head worn AI system that allows children who are born blind or low vision to understand, dynamically, who is around them. And people around you often change and change quickly, particularly in a school setting where you’ve got lots of children who move very quickly. We could really help them to build up those skills of understanding, like learning to create a map of who was around them, then understanding how they could use their body to engage that map.

And I think, perhaps something I see now in a number of times with children that we’ve worked with was, we’ve always co-designed with children throughout this whole process, is not so much about them being receivers of information, but them having that experience of spatial audio as a representation around them to use that experience to realize that they have power in the interactions with others, but rather they can go out and find somebody that they want to play with. They can choose not to talk to somebody by simply learning to turn their head away from those people. And the moment they understand the agency they have in those situations is the moment that we often see a significant change in their ability to place people and, and to engage with them. 

Roberts: So, explain the hardware involved here with PeopleLens.  What is it that someone is wearing and how does it work?

Morrison: Sure. Well, the hardware that we use now, which is the hardware that we hope will become publicly available soon, is a pair of smart glasses. We use the Nreal smart glasses, which have what they call six dots. So, this is able to track the direction of your head in six axes. This is very important for us to be able to give very dynamic feedback. We have developed a head mount, which rather than putting the glasses over your eyes, you put it on like a baseball cap and they sit just on your forehead. So, it doesn’t occlude any vision that you might already have, and it doesn’t include your face for when you’re interacting with others, but it allows your head position to be suitably tracked so that you’re, you’re able to engage as you like with the people around you. It’s connected by cable to a smartphone and that’s it. 

Roberts: Great. And so now, the software does what?

Morrison: The key experience that we provide is that when you look at someone, their name is read out in spatialized audio when it reaches your ears. That means if you’re turning your head from left to right scanning around the room, when you look at someone’s nose, they will trigger their name, but if your head keeps moving, then you’ll hear them in the correct position to your head. So, you can create an almost perfect spatial map of the people around you. It’s a dynamic map. So, it’s changing quickly. So, unlike some of the glasses that are available today, which are using images, we keep an underlying model. It’s called a world state model of all the people around you. And we are updating that as people are moving around, both through predictions of where they might be, as well as for any sensor data that we’re able to capture about where people are.

Roberts: Is the goal to continue to wear the headset all the time, or is this just a learning device with the goal to remove it at some point in time?

Morrison: I think it’s never a great idea to be completely dependent on technology and really where we focus the most of our research is really in that scenario to help open up the experiential opportunity of children who are born blind, build up some skills so that they can then use their other senses.

Roberts: Have there been any surprising ways that young people use PeopleLens?

Morrison: In fact, usually what they would do was to figure out where were their teachers, which is actually pretty useful in here in school. Because if you start talking about your teacher, when they’re there and you haven’t noticed, you can get into real trouble. So, we started to see the technology being used in ways that we didn’t expect, but we also saw that there was a real gap. It wasn’t enough to just know where everyone is around you. You had to know how to use it. You had to come to learn, what do these signals mean and how can I use them to achieve the interaction goals that I have?

So, you hear their name in spatialized audio as it’s read out. You also hear when people look at you. And finally, if you’re in a larger open space, trying to find out who’s around you, we help to guide you towards faces.

Roberts: So, you developed this system, the PeopleLens, to help, initially, children with their social interactions, often at school, but at school we have two functions: one’s social, one’s educational. What have you learned about PeopleLens in terms of being able to help children who are blind be able to function better at an educational level at school?

Morrison: A lot of the learning is done in a social context, so they might sit with another child. They might sit in a group of four. They might have small groups where they’re interacting. And these can be very daunting experiences for a child who is spending a lot of effort just trying to figure out who’s there, what are they doing, what’s happening around me, and very difficult to then actually focus on whatever it is that they’re talking and learning about. When we look at the workplaces of today, they’re often very collaborative places.

So, you can be the best mathematician in the world, but if you struggle to collaborate, you’re not going to be building the AI technologies of tomorrow. And I think by helping kids ensure that they have a very strong foundation in these attentional skills, we’re giving them a significant lift. It’s not something they’re tested on, but when they try and enter the workforce, it’s going to be an important part of whether they’re going to feel comfortable and they’re going to succeed in the workforce.

And finally, one of the things that we’re still gathering evidence about, but we have strong indications of in our current studies, is the building up of attention. Attention is the basis of almost everything we do. And it’s often built, and joint attention being that word that’s used, in the way we interact with people before it’s then built up into a more cognitive function that we use within education.

But many children who are born blind, often struggle with attention. And if you struggle with attention, then you’re going to struggle with any of the educational elements that then sit on top of attention. Whether that’s deciding whether you’re going to do your biology homework, or your English homework, or whether that’s trying to follow a conversation in which there are two or more people talking. Or whether it’s just to attend to the person talking as opposed to the lawn mower outside. For many people, they find these technologies helpful more because they take the cognitive load off of spending so much time just tracking everyone around them. And it just gives them space to then focus their attention on other things.

Akselrud: Once we gain that trust, we can go into the classroom and we can work on role-playing with other children. So, taking turns, having a student verbally speak out the instructions so that the child can take a turn with another child. And getting them used to working with other children. They may just be fearful of even engaging with other children for fear of what the child may think of them, for fear of failure. And, by building up their self-esteem in those social situations, so starting at just engaging with another child on a very basic level, that builds up their self-esteem so that they can try and engage even more with others and be more confident in their ability to engage with others. 

Weiler: When we have Beautiful Lives Project events, children are able to make friends with other children who have other disabilities, also with the hosts of the event or with the other caregivers who might be there. They’re just able to expand their social skills and their social set through being able to take part. It’s a similar desire for the adults as well, because no matter what age someone is at who has a disability, it can always be hard to find friendships, and they can still find those connections through Beautiful Lives Project events. 

Roberts: Greg Stilson and Dr. Morrison work inclusively. That is they work and co-design with people who are blind to create technology that they need, and most importantly, want. And it delivers. And tools like PeopleLens are engaging young people who are blind, not just to lighten their cognitive loads, but to learn the social skills they may otherwise struggle to develop, setting them up for a lifetime of meaningful learning and contribution. 

Finally, not only do we have these amazing minds, dedicating time to inclusively creating tech to make life better for children who are blind, but Dr. Morrison’s story shows us that now the largest tech companies in the world are investing in projects like these to find new applications for the AI that will power our futures.

We’re at a place where technology for everyone is converging with the specific needs of people who are blind or low vision. And that will have huge benefits for students, graduate students and people in the workforce alike who are blind. 

Morrison: I don’t feel the best solution for the blind and low vision community is developing bespoke technologies. Often, they need to be bespoke in the beginning to try them to see what works, but the more and more stuff that we can build into when you show up in your workplace, the tools you need are already there. You don’t have to spend, you know, X amount of money on this bespoke device and that bespoke device and this bespoke device.  Ultimately, you want it built into the tools that you have.

Roberts: What a game changer! What a life changer! What a difference this is going to make in children’s lives. What else is possible? What’s next?

Did this episode spark ideas for you?  Let us know at podcasts@lighthouseguild.org.  And if you liked this episode please subscribe, rate and review us on Apple Podcasts or wherever you get your podcasts. 

I’m Dr. Cal Roberts.  On Tech & Vision is produced by Lighthouse Guild.  For more information visit www.lighthouseguild.org.  On Tech & Vision with Dr. Cal Roberts is produced at Lighthouse Guild by my colleagues Jaine Schmidt and Annemarie O’Hearn.  My thanks to Podfly for their production support.