The Latest Frontier in Tactile Technologies

On Tech and Vision Podcast with Dr. Cal Roberts

Close your eyes.  Raise your hands. Reach out and touch the nearest surface. What are you touching? A desktop, a leather steering wheel cover, a porcelain cup, a plastic keyboard? Our sense of touch and the way in which we interpret the materials in our environment are fundamental to our experience of the world. This episode’s big idea is the new developments in tactile technologies. You’re probably familiar with one of the oldest technologies, Braille, which was invented in 1824 by Louis Braille, a Frenchman who was blind by the age of 3.  Braille, which has undergone numerous refinements since its invention has led the way in helping people who are blind read, write, and interact with the world around them. But as useful as Braille is, it has its limits: Braille is used for text, it can’t convey images.  Two individuals who are working to develop technologies that will one day help people with vision impairment to experience images and graphics is material scientist Dr. Julia R. Greer from Caltech and physicist Dr. John Gardner from Oregon State University.

Podcast Transcription

Greer:  We were developing one of these gels and sometimes it has the consistency of honey or a jellyfish.

Roberts: This is Dr. Julia Greer, materials scientist at Caltech.

Greer: And they’re always polymers.

Roberts: In her lab, Dr. Greer uses 3D printers to design the lattices of atoms.  And the shapes of the lattices determine the characteristics of a material, whether it’s squishy or hard or porous or water soluble.

Greer:  And one of my graduate students stumbled upon this one synthesis.  He applied an electric field to it and then he watched it expand and then he said, wait a minute, this is kind of new.  It was truly what you’d call a scientific discovery.  I don’t think he was looking for that effect.

Roberts:  I’m Dr. Cal Roberts and this is On Tech & Vision.  Feel around you.  Are your hands on a leather steering wheel?  A porcelain coffee cup?  A state-of-the-art keyboard?  Materials are foundational to our experience of touch and to tactile technologies.

Tactile technologies have been around at least since Louis Braille.  Blind from the age of 3.  Invented the Braille language back in 1824.  The big idea in this episode is that the tools and technologies to create dynamic Braille and tactile graphics are innovating an evolving at the fastest pace in history.  And, due to advances in Materials Science, they may be primed for another huge breakthrough.

Computer Voice:  On the Y axis we have diffusivity, and on the X axis we have distance from the source.

Gardner:  Most blind people still don’t have access to good graphics.  The way that graphics are “made accessible” mostly is by audio description of some kind.

Roberts: Dr. John Gardner lost his vision in 1988 at the age of 48 due to complications from a routine eye surgery.  And overnight his life was suddenly different.

Gardner:  You can’t say, this is a picture of a distorted sign curve and really have somebody understand what it is.  They’ve got to be able to see it with their fingers.

Roberts:  When he lost his sight, Dr. Gardner was a Physicist.  A Materials Scientist at Oregon State University.

Gardner:  We put nuclear, radioactive probes into our samples, and as they decayed, we would record the emissions.  The data came out as plots of counts versus time and they formed a sort of a wiggly curve.  We fit that curve.  But, when I woke up one morning blind, I couldn’t see these curves and I couldn’t tell whether there were nuances of the fit.

Roberts:  Dr. Gardner’s graduate students were still learning the physics.  They couldn’t interpret his results for him.  There were only a few people in the world who could help him, so he faxed them his research.  They’d interpret it and send it back.

Gardner:  Not having access to that graphical information was a major problem in carrying on my research.

Roberts:  But he did carry on for the better part of a decade.  He was not just doing physics, he was also testing and designing makeshift ways to plot data in a way he could read with his hands.

Gardner:  There was something called swell paper that you can make things swell up, but it’s limited.

Roberts:  Swell paper is heat responsive.  You print on it, pass it under an intense light and where there’s ink the paper rises so that it can be perceived by touch.

Gardner:  It wasn’t anywhere near good enough for the kind of graphical information I needed to get.  We looked at some additive polymer technologies that would deposit things on paper.

Roberts:  He’s describing something like a 3D paper printing onto paper.

Gardner:  They sort of worked, but anytime that the thing we were making got tall it would just break off the paper.  So, we eventually went back to embossing and we had to develop a much higher resolution embossing format.  One of my students and I invented a new technology.

Roberts:  Boy, did they ever.  Dr. Gardner and his students made incredible improvements to Braille embossing technology by designing the Tiger Software Suite that lets users easily translate text into Braille.  And he refined the ability to print high resolution tacticle graphics alongside Braille in a 20 dot per inch tactile greyscale.

Schading:  I have seen it at work and it is wonderful.

Roberts:  That’s Audrey Schading talking about the Tiger Software Dr. Gardner’s team invented.  Audrey taught at Lighthouse Guild for nearly 25 years.  Courses like Braille, Graduate Equivalency Degree and English as a Second Language to students who were blind or visually impaired.  Audrey has been blind all her life.

Schading:  A Braille book is very expensive and takes up a lot of space.  An Algebra book I remember seeing someone have was maybe 50 volumes.  Fifty!  And if I’m a teacher I’m putting myself in this frame of mind with all the things that I would need and that would include a Tiger.  I would be able to print out the page, emboss that for my student and they would have all the graphics.  And I, as the teacher, would have all the graphics, too.

Roberts:  Necessity is the mother of invention, so they say, and when Dr. Gardner put his energy towards making the tactile graphics technology he needed for his physics, he changed the game for many users who are blind.

First, a distinction.  Braille is a text-based language that relies on touch where users decipher letters and words by moving their fingers along a line of raised dots.  Tactile graphics are non-text materials that can be perceived by touch.  Think of charts, graphs, maps and other information.  Remember Denna Lambert, the NASA Project Manager who lost her sight to congenital cataracts who we introduced last season?  Let’s play some of that tape.

Lambert: Almost like our Milky Way, that’s actually what the galaxy is doing.  It’s spinning.

Roberts:  In this video on the NASA website that we’re using with permission, Denna is exploring the book Touch the Universe by Noreen Grice.  A book of tactile images from the Hubble Space Telescope.

Lambert: It’s kind of cool.  It either feels almost like a spider with legs, but you know that it’s a galaxy in space.

Roberts:  What Denna Lambert is demonstrating is that tactile graphics can be used as a complement to Braille and audio description to give learners a better mental representation of the subject matter and of their universe.

Schading:  A lot of times tests which are given to students are given audibly and the graphs are explained.  And in my mind, I need to touch it.  Touching is like seeing it.  Hearing about a complicated diagram is not the same as seeing it, for me seeing as my fingers.

Gardner:  Nowadays, tactile is becoming more popular.  Still, very few blind people have enough access to tactile graphics to be able to read them so we have technologies combining audio so if you touch a map of the United States it will tell you which state you touched and so forth.

Roberts:  Tiger Software is good, but Dr. Gardner says it’s still not good enough.  One gets the sense from talking with him that he won’t rest until he finds a way to deliver the best tactile graphic technology to users who are blind.

So far we’ve introduced a few tactile concepts.  Braille books, which are great but can take up a lot of space.  Braille embossers, which allow for dynamic printing of Braille, but leave out other kinds of tactile graphics.  And, Braille and tactile embossers, which, thanks to Dr. Gardner can do both.

Some listeners might be familiar with refreshable Braille displays.  These allows users to download dynamic content like eBooks in Braille.  Some can even be used like computers.  Here’s Audrey.

Schading:  I just saw my email from you on that, and as soon as I saw that Braille email I went right to my phone and found the email also on my found and clicked on the Zoom link and here we are.

Roberts:  However, refreshable Braille displays are often quite expensive.  Why?  It has to do with the technology that moves the pins and to understand that, first we must rewind.

All the way back to Paris, 1880.  French Physicists and brothers Jacque and Pierre Curie (Pierre has not met Marie yet) are working on something.  They suspect certain crystals can exhibit an electrical charge if you put the right kind of pressure on them.  Imagine these young scientists in a room applying pressure to crystals using household tools.  They’ve got tin foil, glue, wire, magnets and a jewelers saw.  They do a number on various materials.  Quartz, topaz, cane sugar, Rochelle salt and tourmaline.  And what do they find?  Their hypothesis is correct.  When these materials are compressed, the mechanical strain does result in an electric potential.

What is the point of this story?  Well, let’s jump back to present day.  The piezoelectric effect has a lot of uses including in sonar and ultrasonic transducers first used in submarines and now in our modern-day cars.  Guitar pickups – that part of the electric guitar that converts the sound of strings into an electric signal for amplifiers – those use the piezoelectric effect.

The piezoelectric effect is used in refreshable Braille displays in a particular way.  Each cell in the Braille display is controlled by a piezoelectric actuator that sits underneath it.  When voltage is applied the pins of each cell move up or down in response to that voltage, allowing users to read dynamic text.  But that’s not the end of the story.

Schading:  I’m not an expert on Braille displays but I’ve learned enough to understand and typically they use this piezoelectric type by more actuators.  And they work very well but they’re also very expensive.

Roberts:  Refreshable Braille displays can cost anywhere from $1000 to $8000 or even more.  And, Audrey says, the pins in those refreshable Braille displays can be fragile and expensive to replace, so a more affordable solution to refreshable Braille that also combines tactile graphics would be huge.  Before we get into that innovation, let’s get to know Dr. Greer.

She fell in love with Materials Science in a crystallography class she took for her PhD.

Greers:  What?  Atoms are locked in positions and their lattices, and little did I know that my subsequent research was going to be all on lattices.

Roberts:  Lattices are the patterns that atoms make inside a material which give the material it’s properties.  And, by designing her own lattices…

Greer:  You can get entirely new properties, entirely new functionalities by controlling actual chemical bonds or controlling something we call microstructure, and that is the specific atomic arrangements within a material.  And then, all of a sudden, it has vary unique properties that you hadn’t anticipated at all.

Roberts:  This is how Dr. Greer’s student, who we mentioned at the top of the show discovered what they’re now called electroactive polymers.

Greer:  As he dug more and more deeply into the specific polymers that he was using, he was discovered that there’s some oxygens.

Roberts:  A series of lattices made of oxygen atoms.

Greer:  Then some nitrogens.

Roberts:  A series of lattices made of nitrogen atoms.

Greer:  So, there are two different networks of polymers.  And when you apply an electric field to them it drives them apart because one of them goes to the positive end and the other one goes to the negative end.  And because that drives the polymer chains apart it actually expands the material.

Roberts:  And what are these electroactive polymers like?

Greer: Imagine a small sugar cube or something like a Starburst shape.  It was like that.  And then when we put the little electrodes on the top and the bottom you can watch it expand.  It’s really cool.  It’s slow.

Roberts:  A moment of discover.  One that evokes the moment 140 years earlier when Jacque and Pierre were banging on materials and trying to figure out if they could elicit and electrical response.

Greer:  And the more we thought about it the more we realized this is actually a robust mechanism that is much better than the piezoelectric response.

Roberts:  Could electroactive polymers like the ones Dr. Greer has discovered in her laboratory replace the piezoelectric Braille displays made possible by the Curie brothers?

Greer:  I would love for them to change the game for the refreshable Braille displays.  We’re still working it.  We’re a research lab, so as much as I would love to see it happen, it’s still very much at the development level.

Roberts:  But Dr. Greer has enlisted Dr. John Gardner to help her think of uses for the new polymer gel her team has invented.

Greer:  He explained to us how blind people have access to information.  So it seemed to us that it was just a very natural connection between having this poly-electrolyte gel that swells in response to an applied field.  You don’t need to submerge it in any liquid or any additional solvent.  you don’t need to use any complicated fab techniques.  Now that we have it we can shape it into a 3D shape of some kind and then apply the electric field only to those points where we need them to be.

Roberts:  A Braille display only displays Braille.  But an electroactive polymer display could take data inputs electronically and switch between Braille and tactile graphics like charts and maps, and back to Braille again with the same ease that sighted people switch from pictures to text on their computers.

For their research, Dr. Greer and her team have interviewed a number of people who are bling or have vision impairment.

Greer:  All the ones that we’ve interviewed said we wish we could see pictures, or we wish we could see the data.

Roberts:  Close your eyes.  Imagine you’re at a science lecture and the speaker stands in front of a visual presentation that you can’t see.

Greer:  On the Y axis we have diffusivity, and on the X axis we have distance from the source.

Roberts:  But then the speaker never describes what’s happening in the plot.  They just assume everyone can see the distribution of data and can infer the story that data is telling.

Greer:  I was just thinking the one application where I can readily see these electroactive gels is that we can just apply the electric field along the shape, along the plot, whatever feature it is you’re trying to convey because that’s not a hard thing to do.  It would be like an iPad that can also raise from the plane where it’s at.

Roberts:  What would a listener in the audience experience if they had vision impairment and were able to perceive graphs through an electroactive polymer display.

Greer:  They just can feel where the points are.  Oh, so these have a large spread, or these are very tightly clustered together, or there’s a line that shows a particular behavior that it first rises and then it falls, or something like that.  Those are the kinds of things we all take for granted because we have eyesight.

Roberts:  This is the exact problem that Dr. John Gardner had in his own lab after he lost his vison.

Greer:  It just seems like such a meaningful research pursuit.

Roberts:  While it may not be hard as Dr. Greer said, it certainly won’t be easy.

Greer:  So, of course the main challenges are the response time.  So, if your gel takes an hour to produce that shape, it’s not very useful.  And then of course, the sensitivity.  What is the human touch – how much does it need to be raised by?  If you say apply 40 volts per centimeter how much of an expansion can you expect.  Well, if it’s 10% per minute, is that something that your finger will be able to detect?  No, so there’s still quite a few challenges that have to be worked out, but that’s what makes this problem so interesting.

Roberts:  What technological innovations allowed for the kind of materials research Dr. Greer and her students are undertaking in her lab?

Greer:  The one big investment that’s worth mentioning is called additive manufacturing.  What that means is that we now have equipment like 3D printers, very sophisticated 3D printers which is what we have, which allow you to sculpt effectively, voxel by voxel exactly the sculpture that you want.  So, that’s a necessary condition.

Roberts:  We love to say that science doesn’t happen in a straight line.  And this is a beautiful example.  Commercial and research 3D printers not developed for people with blindness or vision impairment enable Dr. Greer to design new materials on the atomic level, constructing the lattices that change material properties.  And, by doing this, her team has developed the electroactive polymers that could change the game for learners, researchers and scientists who need tactile representations of otherwise visual data.

What else could electroactive polymers be used for?

Greer:  Anything that has a pictorial depiction, be it from architecture to science to engineering to simple art.  Think about it.  Our blind colleagues can’t see art.  So, to be able to take some Rembrandt – there’s nothing that’s stopping us from doing that.  I personally think that the power of this technique is in the dynamic change.

Roberts:  Remember, refreshable Braille displays allow for dynamic content but only in Braille.  Electroactive polymer displays would allow for so much more.

Greer:  You can see data first, and then you would see a schematic.  But, even watching a movie, eventually you could imagine that if this technology were to come you could feel what’s happening.

Roberts:  And the electroactive polymers Dr. Greer is developing could have many uses beyond Braille and tactile displays.  These polymers could replace the piezoelectric actuators in the guitar pickups we talked about earlier.

Greer:  Any time you have a robot or a drone or an actuator or a sensor that uses these kinds of piezoelectric materials they’re typically not cheap.  In contrast to what we are developing here, it’s a polymer so it’s already a lot more compliant.  It’s pretty simple.

Roberts:  Before we get too ahead of ourselves though, change takes a long time in Materials science.

Greer:  Probably more like ten years.  The response times is something that really needs to be worked out.  I think that because our society is so familiar with the piezoelectric materials and they’re very well-versed in that, there’s no need to replace the piezoelectric market, or even to venture into that.  But there’s certain applications where this kind of gel would do such a much better job.

Roberts:  Well, if ten years seems too long to wait, let’s remember that the piezoelectric effect wasn’t used for much of anything until 30 years after it was invented when it was first put into submarines for sonar.  While we wait, Dr. Greer, ever the materials scientist, is excited to get back to testing her polymers.

Greer:  That’s the most exciting part to me.  How much can we push them apart?  What are the actual limits there?  Right now, everybody’s dealing with these one micron, or 40 microns or 20 microns is what the hand can sense.  Maybe we can go 100 microns.  Maybe we can go to half a millimeter.  For me, the most exciting part is the discovery of a new mechanism in that what can you do with it.

Roberts:  What does Dr. Gardner think about electroactive polymers.

Gardner:  Well, I’m a material physicist.  The technology she’s talking about needs a lot of development but potentially is something that could be just perfect and could be quite inexpensive.  And I think that’s pretty exciting.

Roberts:  Audrey agrees.

Schading:  I can send it to the student, they download it and it’s on their display and they have this page in front of them that’s electronic.  So they’re going to be able to touch these graphics on their display and they’ll save them so they don’t even need the paper then.  It’s going to be all electronic.  So, that is the next step in the evolution.

Roberts:  Dr. Gardner wants tactile feedback in more places.  He reminds us that tactile feedback is already working its way into products for sighted people in ways that could benefit people with visual impairments.

Gardner: Mercedes is building tactile sensors into their steering wheels to help drivers.  Because, when you’re driving your senses are just overloaded and you just can’t keep looking down at your dashboard and doing everything visually.  You’ve got to have audio and tactile feedback.  The audio is overloaded as well because people like to listen to music or news or whatever on the radio.  So, tactile output has potential.  Fairly major and fairly important potential.

Roberts:  For her part, Dr. Greer sees herself advancing materials science through inquiry and testing.

Greer:  There’s no magic with it.  It’s a random discovery.  So that’s a little bit of magic,  But then it’s just hard work.  It’s hard work to explain it, it’s hard work to think through it, it’s hard work to fail.  You keep on failing and failing and you just get up and keep on thinking about it and you don’t give up.  It’s a platitude and easier said than done, but that’s really what we do.

Roberts:  Some day, tactile graphics for charts, graphs and maps could be as readily available and dynamic as Braille is today.  And dynamic Braille displays that integrate tactile graphics may be on the horizon.  How will tactile feedback and haptics integrate into the technologies of the future?  Anything that inspires students who are blind or visually impaired to become the researchers, scientists and technologists of tomorrow is very important to us at Lighthouse Guild.  And what science will they produce?

I’m Dr. Cal Roberts and this is On Tech & Vision.  Thanks for listening.

Did this episode spark ideas for you?  Let us know at podcasts@lighthouseguild.org.  And if you liked this episode please subscribe, rate and review us on Apple Podcasts or wherever you get your podcasts.

I’m Dr. Cal Roberts.  On Tech & Vision is produced by Lighthouse Guild.  For more information visit www.lighthouseguild.org.  On Tech & Vision with Dr. Cal Roberts is produced at Lighthouse Guild by my colleagues Jaine Schmidt and Annemarie O’Hearn.  My thanks to Podfly for their production support.