Ambient Computing and Voice Assistants: From Your Home to the Stars

Lots of people have voice-controlled smart home assistants like Siri, Google, or Alexa in their homes….to listen to the news or to set timers. But they can do so much more! David Frerichs, Principal Engineer, Alexa Experience at Amazon on the aging and accessibility team, shares his design philosophy for making voice assistants more inclusive, and the preferred mode of engagement for every user. He also shares that the next stage of smart home assistants will be ambient computing, where your devices will intuit your needs without you speaking to them. We talk with Lighthouse Guild client Aaron Vasquez, who has outfitted his home with smart home technology, and with Matthew Cho, a client who traveled to the Johnson Space Center in Houston to speak to the unmanned Orion Spacecraft via the Amazon Alexa on board, demonstrating that voice assistant technology can bring inclusivity and accessibility to many jobs and industries and are not just for the home anymore.  

Podcast Transcription

Cho:  My name is Matthew Cho and I’m 18 years old. I became blind beginning of sophomore year in high school. I had a brain tumor and I had two surgeries to remove it and my second surgery I lost my vision. So, my whole life changed. I miss having my vision, but it’s been like a rollercoaster ride all the way. 

Roberts: Over 25 1/2 days, from November to December of 2022, NASA tested in deep space and unmanned spacecraft named Orion. This was the first in a series of missions to their part of the project Artemis, aimed at building a long-term human presence on the moon. The Orion spacecraft also hosted an Amazon Alexa device through which Mission Control, located within NASA’s Johnson Space Center, tested their ability to direct the spacecraft with voice commands from Houston. 

When Amazon sought volunteers to join the virtual Artemis crew to test the Alexa on board the Orion, they wanted to find a volunteer to represent people who are blind or visually impaired. To this end, they reached out to Lighthouse Guild. We chose Matthew. He flew to Mission Control in Houston in December, and Lighthouse Guild’s marketing manager, Jeremy Morak, spoke to him in the minutes before he got to communicate with the spacecraft. 

Cho: I don’t know. It just feels like I’m in a totally different world, like I’m in outer space. It’s just a wonderful opportunity. I feel that just listening, feeling the actual materials that they use. It reminds me that it’s really just so wonderful that we can use technology to drive the spacecraft on its own. 

Morak: And so, what is just one thing that you think  you’re going to want to ask Alexa.?

Cho: I would have to say definitely asking Alexa to play some music while in space. Maybe, I don’t know, like probably something like maybe like some country music. To be able to have a rocket be dependent on an AI without anybody having to control the spacecraft is really amazing and I feel that later on that they’ll be able to use it for much more things aside from space. 

Roberts: The use of Alexa in the Orion spacecraft is one small step for Amazon and NASA, and it represents a giant leap in the way that people who are blind or visually impaired engage with their environments here on solid ground. Today’s big idea is not just voice assistance but ambient computing. Ambient computing describes a network of sensors in the built environment that anticipates users needs and responds accordingly.

This would not just benefit people who are blind but may be the preferred mode of engagement for any user, regardless of ability. The use of Alexa in the Orion spacecraft is a first step in this direction. 

Cho: So, their goal was to eventually get different people to be able to go into space. And one thing that they wanted to help that mission happen would be to introduce AI into space into missions like Artemis and they were trying to see if Alexa would work properly with all sorts of people, normal people. Not just astronauts. Like regular, ordinary, everyday people. 

Roberts: David Frerichs is principal engineer of the Alexa experience at Amazon. 

Frerichs: Looking back at what we’ve done with regards to the NASA Artemis One mission, our participation in that with Lockheed Martin has been to really understand the challenges of voice interactivity in the space environment, not only from a physical point of view, and that’s where Lockheed Martin really did a lot of the work on that front, but also in terms of the acoustic modeling and other kinds of interactions and the lack of connection to the Internet on Earth, right? 

You have all these different things that are going to push the boundaries of improving speech recognition and wake word recognition. You’re going to have the ability of improved functionality for local interactivity with Alexa versus relying on the cloud. These are all capabilities that are being explored that are going to have a much broader potential impact. 

Just like if you think back to how Velcro was created for Apollo, right? The hope is that by investing in this research and by putting this payload into the Artemis One mission, we’re going to similarly have a benefit in understanding of how to use voice interactivity to improve astronaut performance when they’re on missions. 

Roberts: So tell us, how did you get to making Alexa more accessible? 

Frerichs: My current role at Amazon Alexa, working in the aging and accessibility team, is a culmination really, of a philosophy that I’ve had for the past 30 years. I like to make machines that conform to the needs of humans. And that is really the way that things should be. Too often we accept the idea that the person has to adapt themselves to use the machine better. And yes, it is true that even the most simple machines, and I’ll call a pencil a machine for example, takes a while of training to learn to use it well.

Now that we have this ultimately flexible tool called the microprocessor, the most widely manufactured tool in human history, that has the ability to adapt itself at the instruction of people for people. You can actually now make it so these machines become more conformant to the way that a particular customer might want to use it. And so, that is something that I’ve been working on first in virtual reality, then in connected devices and Internet radio moving forward into voice agents and most recently in an ambient computing, and now most recently in my role at Amazon. That’s been my thread the whole time. 

Roberts: Often I’ll go and I’ll speak to a donor and I’ll demonstrate some kind of advanced technology, maybe something that does really amazing facial recognition. For example, the donor says, hey, I don’t know about the people who are visually impaired, but I’d love one of those for a cocktail party that I knew who I was talking to. 

Frerichs: Exactly, exactly. 

Roberts: And often what we find is that technology that was designed primarily for people who are vision impaired, boy it can roll over and be something for the general population.

Frerichs: Absolutely. And it really comes from an idea of how you look at barriers that people have using your product and then how you address those barriers. And Microsoft came up with a very interesting model that I have used and then actually expanded upon to really look at these kinds of barriers that are there. 

You have permanent barriers, which I think are things that people are typically associating with an accessibility use case. But then you also have temporary barriers, things that may be affecting someone on a temporary basis because of something that they are encountering in their life at that particular moment. Then you go beyond that to a situational barrier. It’s a blocking because of the fact of where or when the service is being used. 

And then the thing that I’ve added on top of that from the Microsoft model is something I call preferential barriers. That is to say, I would rather use the product this way than that way. So if you think about the origins of curb cuts, it comes back from activists like Judith Heumann back in 70s really trying to make it so that people who were using wheelchairs were able to safely cross the street. And that is something that really was a permanent need on the part of that community. And they addressed that need. 

But then you look at, OK, what about a temporary need? I’m somebody who just got out of the hospital or I’m somebody who has a baby. In the UK they call them pram cuts for a reason or situational, right? I’m moving and I have a hand truck. But then the thing that’s really cool is there was a study that was done in Sarasota, FL that showed that nine out of 10 people go and use a curb cut if it exists. Why? To me that’s a preferential use case. They just want to do it. It’s easier and that’s what they do. 

So, we have this four-level model of permanent need, temporary need, situational need and then preferential need that really can inform us on when we see a barrier, how can we address that barrier for the particular core use case, but if we do it well, it will actually serve a much broader community. 

Roberts: And so there are two types of needs often. There are what we call unmet needs and then there are unanticipated needs, and so that someone like yourself can see an opportunity that the end user never even thought of. We get back to this whole idea of the smartphone. Before Steve Jobs, no one ever thought about what else you could do with the phone other than make a call with it. 

Frerichs: Nokia might disagree, but OK. 

Roberts: Exactly. And so, he had a vision that no focal group or no advisory board would have ever given him. And so, I’m trying to understand in your world, you know, how that mix is. 

Frerichs: So, the job of a good product designer or a good technologist is to enumerate a vision of what could be that meets the customer’s expectations but that may be unexpected. But definitely is designed with that customer in mind. I think the iPhone itself is a perfect example of that. There was not a single piece of technology in the iPhone that was actually new. It was a reimagining of how to use those technologies together in such a way that serve the customer need in a way that they didn’t even anticipate until they saw it and now it’s part and parcel to our whole existence of the society, the way that we interact with these phones. 

And so, I think it’s that combination of, OK, understanding what’s possible, seeing all the pieces, fitting them together in a way that is done with a deep understanding of the customer. That’s the genius that we’re looking for every single time. 

Roberts: And one example of that might be what you’re doing with natural language processing where you understand what’s possible and then you think about, boy, how much would the customer truly benefit from that? 

Frerichs: Absolutely. And you know, the idea of voice interactivity with computing devices is something that is going way back in our science fiction history for quite a long time. You can go way back, all the way to the origin of the word robot, the idea that I have a machine that I can speak to or going back to the golem right there we go all the way back to the idea that there is an automata that we can speak to and then it will respond to us in such a way. And we’re very naturally verbal. And so, to make it so that a machine can understand what we say and then be able to enact and act upon what we say is a very powerful mechanism. That, of course, is part and parcel of what we do in Amazon Alexa. 

As a sighted individual, sometimes I can have a hard time really internalizing directly the needs of the blind and low vision community. So, I have to work with people who are actually blind and low vision in the product design process, which is what we do. But another thing is that when I’m doing it internally, one of the models that I think of is what I call hot tub safe computing. That is to say, if I have my eyes closed and my hands are wet, could achieve everything I want to achieve with the machine? And so, that really puts some Christmas on. For example, one thing people don’t think about the length of the response of the machine, right? Is it pithy? Is it getting to the point or is it overly chatty, which can be super frustrating to people that use audio interfaces. Or, can I just say what I want as opposed to having to go to a smartphone and achieve what I want, having to touch a smartphone to do it? 

Roberts: As David boosts the accessibility of the Amazon Alexa, people with vision impairment are increasingly integrating it and voice assistants like it into their homes. 

Torres:Good morning.

Alexa: Good morning. Let’s get ready to start the day. Turning lamp on. Opening shade. Turning temperature to 72 degrees. Turning on TV. 

Roberts: Monica Torres demonstrates the Amazon Echo device in Lighthouse Guild’s Technology Center. People with vision impairment can ask Alexa, 

Torres: Echo. What am I holding? 

Alexa: Let’s see. It looks like Cheerios breakfast cereal.

Roberts: Or…

Torres: Echo turn on oven. 

Alexa: OK.  At what temperature?

Torres: 350 degrees.

Alexa:  Preheating oven to 350 degrees Fahrenheit. 

Frerichs: Alexa today is a very powerful tool for people to interact not only with information but also with their physical environment. I think the idea of being able to simply ask the device a question and it will answer you, the ability to ask the device to turn on the light and, if you set it up well, it will know that the light is the light in the room that you’re asking about, not the light in the kitchen. The key thing that we’re moving towards is something called ambient computing. That is to say that the system should be able to respond to the needs of the customer even if the customer doesn’t say anything. 

Sensing the environment, sensing the context, being able to then provide an ambient computing model that really takes care of the customer overtime and really serves the customer well. That’s something that I think is where the boundaries are and where it’s going to continue to be pushed. That’s the first one. 

The second one is the personalization. That is to say, how does it serve an individual? What is David doing right now? How can I best, as the machine, help David out with his goals and what he’s trying to achieve, and then specifically when David interacts with it, how can I customize my interactions? Those are the kinds of things that to me, that we do today that do point the way to the future. We look at the opportunity space of what I was talking about before, which is the ambient computing solution taking inputs from multiple devices and synthesizing them into a model that then the agent Alexa can use to help out the customer, whether that’s there’s the stove, which is at a certain location from where you’re standing right now or any other kind of interaction guidance that someone might need.

Roberts: That’s great. Smart Homes is one area that’s really important to us, the Lighthouse Guild. We built a smart home within our technology center just for the purpose of helping our clients and our patients just experience what it would be like to have a smart phone. 

Frerichs: Absolutely. 

Roberts: And to take away the fear factor because, though it’s getting better, as the baby boomers age and become the older population, we still see a certain group of our visually impaired and our elders who are scared of technology. And so, what are goals, what are object strategies to help remove the fear factor from people who could really benefit from technology, but aren’t getting there yet. 

Frerichs: My experience – and I do a lot of work with older adults as I mentioned – my role is in both aging and accessibility in Amazon Alexa. The thing that I have them articulating are concerns over privacy, for example, or articulating frustration at the wonkiness, I’ll call it, or the fidgetiness of technology. It’s more of making sure that the product addresses the privacy concerns, and making sure that the product addresses the usability needs of those older adults and meets them where they are. And so, that I think is the key goal of a technology like Alexa to be able to meet them in terms of addressing their privacy concerns, which is something that is part and parcel and one of the main reasons why I like working at Amazon is because of our tremendous focus on privacy.

And then, in conjunction with that, making sure that the product is actually working with them in a way that they expect, not the way that some engineer decides is right. And I won’t claim that we’re there yet in full presence of self-criticality, which is also something we try to do at Amazon. There’s still a lot of improvement we can do in those areas, but I think that’s the way to think about it. It’s not that folks are afraid. I haven’t experienced that myself. It’s that folks want to know that it’s private. Folks want to know that it does what they want it to do, and folks want to know that it’s not going to be some fidgety wonky thing that they’re going to have to learn some esoteric set of commands to interact with.

But the idea of where we’re trying to go is to make it so that an older adult or any customer can speak to the device in a way that’s most comfortable for them and have it do what they expect it to do. That’s the goal. And when we achieve that goal, that kind of reticence to using the technology will diminish significantly. 

Roberts: So, what did Matthew Cho do inside of Mission Control at NASA? 

Cho: So, one thing I asked was “How fast are you traveling?” And it was really just so crazy. Alexa responded with “I’m traveling like 2000 mph.” It’s pretty crazy. 

Roberts: He asked Alexa a few other pre-approved questions, and then because music is such an important part of the Matthew’s life, he asked Alexa to play a song. 

Cho: I also asked it to sing me a song. And all the sudden it started to sing a song. It wasn’t like a song like from I don’t know, let’s say like Taylor Swift or Justin Timberlake. It was random song that she made up.  It was funny.

Roberts: Asking Alexa to play songs and guitar tutorials on YouTube was an early way that Matthew came to terms with his vision loss. 

Cho: I loved music ever since I was young. I played the piano when I was younger and when I lost my vision I was really like just, I’m so depressed because it was right during COVID when COVID started and I had a guitar then and I didn’t really know how to play it then, but when my mom handed me the guitar, what I’d do is I played YouTube videos. I would search guitar tutorials and then yeah, you know. I just listen to you videos and I kind of learned it on my own. 

Roberts: One of the great opportunities for natural language processing is to decrease the amount of training that the user potentially needs. Because, if you have to learn a certain set of specific commands in order to use the technology, then the amount of training increases dramatically. However, if you can just speak to the technology in words that make sense to you, now there’s I think a faster adoption of the technology. 

Frerichs: So, there’s not a non-zero learning curve for anything. Yes, the machine does need to conform to the person, but we also need to acknowledge the fact that in any situation, technology is not magic. It’s still going to require some kind of understanding of hey, here are the things that the thing can do, here’s the things that it can’t do, or here’s the way that I talk to the thing so that it can tell me here’s what it can do and it can’t do. There will have to be some of that, but minimizing that and presenting it in a way that’s according to the customer expectation and to me is repeatable and non-mutable is going to be key.

Because working with older adults and accessibility customers, the thing that I’ve seen is most frustrating is not the learning of the commands or learning of what to say to the machine. The thing that I’ve heard and understood that is most frustrating is it did this yesterday and I said the same thing to it today and it didn’t do the same thing. That’s what drives people bananas. 

Roberts: Absolutely. And now, how does AI help the machine that it starts to learn the way you say certain commands over and over again and to try to get a certain outcome from these voice commands?

Frerichs: In machine learning is a reaction mechanism. It’s best thought of, in my view as an engineer, as a really good reflex, right? It’s just like when you hit your knee with a hammer and your knee does the thing. That’s what’s happening in terms of artificial intelligence and machine learning. You’re training it so that you give it some input, and then it will actually give a specific output on the other side. The power of that is that it can start with a large corpus that’s trained on a wide group of folks, but then a well-implemented system will take into account the individual’s preferences and needs overtime, sometimes naturally, sometimes through interrogation, meaning it will do a Q&A to understand the person better and then it will keep that and become contextually aware. That’s really where the power will manifest. 

Roberts: At Lighthouse Guild we are very proud excited to partner with Amazon on the Artemis project. So, tell me about what that means to your group that someone like Matthew can imagine a career for himself in the aeronautics or space exploration field. 

Frerichs: My hope is that this kind of deployment and having someone like Matthew be able to interact with it is going to really inspire folks to say, wow, you really can take a machine and make it adapt to the need of the person. You really can do this. We don’t need to be stuck in our previous metaphors. We can actually open the door and make this kind of interaction available to a much larger population. 

I think that, again, the reason why we remove barriers is because barriers are keeping people who we want to have participating from participating. So if we can remove these barriers it actually empowers us not only on the individual level, but it empowers us as a society to benefit from everybody’s contribution. 

Roberts: So, how was Matthew’s experience as part of the virtual Artemis crew? 

Cho: I just felt very optimistic for the future that hopefully one day that they can use visually impaired employed to – there was another person who came with me on the trip too, and he had this disability, he was on a wheelchair and se was chosen for this. I know there – this is a virtual experience. I realized that he can’t really get up and push the button and that’s why having a Voice Assistant, an AI, it really does open space. Later in the future, one day they took someone with a wheelchair into space it would work, hopefully.

Frerichs: And the thing that I’ve been doing most recently that I’m happy to share about is the collaboration we’ve done with the University of Illinois Urbana-Champaign. In conjunction with Apple and with Google and with Meta and with Microsoft, where specifically we’re forming a coalition that is run by the university to be able to collect machine learning datasets for speech, targeting folks that are in underserved constituencies, for example Parkinson’s, ALS, these kinds of needs. 

Currently, some of them have more of a challenge interacting with voice agents in general, not just Alexa, but with all voice agents. Instead of doing something individually and trying to create something that was just for one solution, we decided to work together and have the university collect data sets to be able to serve a wide range of groups. And so that’s an example of the kind of working together in the industry that I’m very proud of and I’d like to see more of going forward. 

Roberts: We’re very appreciative of your time. We love your passion. We’re very excited for what you’ve done and what you’re going to be doing in the future. So, thank you for joining us. 

Frerichs: Thank you so much. I really appreciate the opportunity to be able to speak with you. 

Meir: I’m Idan Meir, cofounder and CEO of Right Hear, indoor orientation technology and I love On Tech and Vision because I always find new technologies that I never heard about them before. It’s also very inspiring for me to see how great developments are there in this amazing field. 

Roberts: If you’re still listening, we assume that like Idan, you too love On Tech and Vision. Thank you, if that’s true. If you like what we do here, it would mean a lot if you could send this episode to two other people you know who enjoy listening and please rate and review on tech and vision wherever you get your podcasts. We are so grateful for your listenership and your support. 

Up next, a Lighthouse Guild client has put smart home technology to use. 

Vasquez: So actually, I was a huge skeptic. I was a huge skeptic because I was on the argument, whereas like, what if your Internet goes out? Then you’re kind of lets your own devices there. You gotta figure it out. I’m Aaron Vasquez and I have a Google Nest audio. And my affiliation with Lighthouse is, I go there for services. I was born with ocular albinism. So I was born with low vision. And detail wise I can see about as far as my hand can stretch. And after that, details drop off significantly. 

Roberts: After receiving a free Google Nest, Aaron quickly got comfortable with the easy use of the device. 

Vasquez: I was like, OK, you know, it’s kind of cool. It’s kind of cool. You know, it helps a lot of little things that you definitely would need. The timers, obviously, those are big help. But since it was also attached to Google, you can make searches with it like for our pets. If we’re not sure if we can feed our cat something, just be like hey Google, can cats eat this and she’d give us an answer. 

And even the weather we use that function every single day. Literally every day. 

Roberts: But the device could do so much more than Aaron and his wife were using it for. Aaron’s lightbulb moment was literally a light bulb moment. 

Vasquez: One day we had gone with her mother and Walmart, and we had seen this pack of smart bulbs, and there were relatively cheap when I was like, oh, that’s kind of cool. So, we’re like, you know what, let’s get it. So, we got them, we hooked them up, and that kind of is what started everything. We have one in our lamp, our main lamp for the apartment, and then we also have a desk lamp that are both controlled over Wi-Fi. Sometimes I forget to turn on the lights before I get home, so when I walk in, it’s pitch black. My cat’s instantly by the door, he wants to get by your legs, and I don’t want to step on him. It’s hard to like, maneuver over to that light switch to go and get it. Whereas I can just say, hey Google, turn on my lights, she’ll turn on the lights, or if you wanted to turn on one specific light you just say, hey Google, turn on like the desk lamp or turn on the room lamp. So, you could just be like Hey Google, turn my desk lamp white at 50% brightness. 

Roberts: Aaron and his wife also use Google to keep a watchful eye on their cats via a pet camera they installed. 

Vasquez: So, if you’re in the house and you’re about to leave you can say, hey Google, start recording on my on my camera and she’ll start the recording. So we watch our cat because our cat likes to get into absolutely everything because he’s only eight months. We watch him live and we can like, talk through the camera. It’s great.  I love that.

Roberts: They use their Google device to watch television. 

Vasquez: So, through Google you can just say, turn on the TV and the Chromecast itself has a chip that can turn the TV on, but it works with certain TVs, so it hasn’t worked with every single TV that I’ve tried it on. 

Roberts:  Though he started as a skeptic, now Aaron is hooked. 

Vasquez: Honestly, so much easier. If I can ask Google a question and she can come up with the answer, then I’m better off that way instead of actually trying to, like look it up myself. It’s cool because she’ll not only read search results as you would on Google, but she would read the articles from websites themselves. 

Roberts: Just because this smart home device is convenient doesn’t mean that it works seamlessly yet. 

Vasquez: So, as the tech is evolving, the way that it understands is getting a little better. It will understand most things. But, if you word things differently, it has a hard time understanding your command. So sometimes I’ll tell Google like hey, turn my desk lamp. Turn it white at 25% brightness. If you chain the commands together, oftentimes she won’t get it. So, it’s a bit annoying because you would want to like, chain them together and have them done at that moment. That’s where it could disconnect is because you have to give it more than one command to get the singular outcome that you want. 

Roberts: For Aaron, the biggest hurdle for smart home technology is that customers have to buy into one company’s ecosystem. It’s kind of finding the right accessories that can be compatible with the ecosystem that you choose. So, there are ones that say, like, oh, works with Alexa, works with Google Home, and that kind of puts people off because they’re like, why can’t I just buy one thing and it be universal? There’s talk for universal standard being created. At the moment, it’s just talk. But I would say within the next couple of years, maybe. So, it would make it more accessible to most people who want to actually get into it, but don’t want to be locked down to one ecosystem. 

Roberts: The voice enabled Smart Home is already a game changer for people who are blind and visually impaired. But if David’s work proves out, smarter and smarter algorithms will anticipate our needs without our even speaking them, then we’ll truly be in the age of full ambient computing. In the meantime, a virtual crew from NASA’s Johnson Space Center has guided the Orion spacecraft using voice commands but has also shown how voice activated AI assistance could level the playing field for people with vision impairment, not just inside of homes, but inside of any professional role or industry from here to the stars. 

Did this episode spark ideas for you?  Let us know at podcasts@lighthouseguild.org.  And if you liked this episode please subscribe, rate and review us on Apple Podcasts or wherever you get your podcasts. 

I’m Dr. Cal Roberts.  On Tech & Vision is produced by Lighthouse Guild.  For more information visit www.lighthouseguild.org.  On Tech & Vision with Dr. Cal Roberts is produced at Lighthouse Guild by my colleagues Jaine Schmidt and Annemarie O’Hearn.  My thanks to Podfly for their production support.