Paul Rand: Every morning when you grab a cup of coffee and take a sip, your brain performs an extraordinary set of tasks. It feels effortless, but behind those simple movements are a complex cascade of signals that are firing across millions of neurons.

Nicholas Hatsopoulos: What people are maybe not aware of is that movement involves a huge part of the nervous system.

Paul Rand: That’s Nicholas Hatsopoulos, professor of organismal biology and anatomy at the University of Chicago. For decades, his lab has been mapping how the brain controls bodily movement, and they’ve used that research to build neuro prosthetics that allow paralyzed patients to control robotic arms or computer cursors using only their minds.

Nicholas Hatsopoulos: So muscles are just different things than motors. And so our brains have evolved to control muscles. And so how could we do this? How could we actually activate muscles?

Paul Rand: Hatsopoulos and his team discovered that they could map all of the little brain signals that happen when an arm moves from left to right or right to left.

Nicholas Hatsopoulos: Based on work that I had done in my lab with non-human primates, when these animals actually watched a cursor move as if they were moving it, but just not moving their arm, those cells in motor cortex would activate as well. But with that knowledge, we use it in humans.

Paul Rand: And his lab is working with people with spinal cord injuries, amputees, and even ALS patients who have lost the ability to move their hands.

Nicholas Hatsopoulos: So it could actually help them with daily living. It could help them dress themselves or feed themselves or whatever they’re doing, and it would just improve the quality of their lives at home or in the world outside of the lab.

Paul Rand: From the University of Chicago Podcast Network, welcome to Big Brains, where we explore the groundbreaking research and the ideas that are changing our world. I’m your host, Paul Rand. Join me as we meet the minds behind the breakthroughs. On today’s episode, the revolutionary advancements bringing sensory touch to artificial limbs. 

If you appreciate hearing from experts in research and science who are shaping our world, please consider subscribing to the Big Brains YouTube channel. It ensures that you never miss an episode and it helps these important conversations reach a broader audience of learners. 

Today’s episode of Big Brains is supported by the Court Theatre, Hyde Park’s Tony Award-winning theatre located on the University of Chicago campus. Here, timeless stories speak to today’s world. From Sophocles to Tom Stoppard, Caryl Churchill to Anton Chekhov and August Wilson to Tennessee Williams. When you’re at Court Theatre, you know you’re going to see something bold and provocative, something that will move you and make you think.

You know you’re going to get a great theatrical experience, unlike anything else, one that only could happen on the South Side of Chicago. Reimagine what classic theatre can be. Visit CourtTheatre.org. 

I wonder if we can start, Nicho, with just really just talking that you’ve spent 30 years exploring how the brain turns a thought into a movement. What in the world has captured you at the beginning and has kept sustaining you over that period of time?

Nicholas Hatsopoulos: Well, I’ve always been interested in movement. When I was a kid, I was interested in Greek dances, dancing. I was also interested in robotics and always wondered if we could build a robot that could dance as fluidly as a person. So that’s what captivated me. And I began doing work in motor control as a graduate student and then continued in postdocs, two postdocs that I did, and now as a professor. And it’s always been just an interest of mine.

Paul Rand: Well, when we talk about paralysis, the thought that immediately comes to mind, at least for me and maybe others, is not walking. Studies have shown people with quadriplegia what they want back most is their hands ahead of walking. Why is it the hands, especially as you explored in the beginning, you can’t dance with your hands?

Nicholas Hatsopoulos: Well, because you use your hands to interact with the world. As human beings, unlike other animals and even primates, other non-human primates who do use their legs for a lot of manipulation, interacting with the world, we don’t really do that. I mean, except in special cases, generally we’re using our hands and our arms to interact with the world. And that’s whether it’s grabbing a cup of coffee, typing on a keyboard, talking on the telephone, anything. It’s so important for our daily interactions with the world, dressing ourselves, feeding ourselves, whereas locomotion is important, absolutely, but there are wheelchairs, there are other technologies that can help.

Paul Rand: Got it. But less so when it comes to using your hands and your arms.

Nicholas Hatsopoulos: Exactly.

Paul Rand: Well, one of the points that stood out to me in doing some research was saying that a hand you can move but can’t feel is close to useless.

Nicholas Hatsopoulos: As soon as you now want to interact with the real physical world and grab things and touch things and manipulate them, you can’t rely purely on vision to help you do that. You really need touch and this other modality called proprioception or kinesthesis to allow you to interact with the world with real physical objects.

Paul Rand: And so when that discovery, if that’s the right word was made, and you think that, well, we’re going to solve the movement issue and you think we’ve done it and then all of a sudden you realize, wait a minute, this is not enough.

Nicholas Hatsopoulos: Well, I think that when we move from just cursor control, that’s how we started. Our first brain gate human subjects back in, first one in 2004, he was a spinal cord injured individual and he was controlling a cursor to surf the web, check his emails, and he used vision and it worked. We definitely, proof of principle it worked, but we soon realized, well, if that’s important, interacting with a computer is important, but that’s not all there is to life. There’s interacting with a real physical world, dressing yourself, feeding yourself, whatever. It wasn’t so much a discovery. It was just a realization that the next step would require this kind of sensory feedback that was missing. So what has happened since 2004 is we’ve developed better algorithms for decoding the brain signals in the brain. We’re increasing the number of electrodes we implant. That has led to now the ability to control robots with many degrees of motion.

So seven degrees of motion, including moving of the wrist, moving of the hands. And so that’s how the field has evolved. And then, I don’t know, maybe almost 10 years ago, about 10 years ago, people at the University of Pittsburgh decided, well, let’s put electrodes in another part of the brain, the somatosensory part of the brain, electrically stimulate that part of the brain to evoke elicit percepts of touch.

Paul Rand: Wow.

Nicholas Hatsopoulos: And now subjects can not only move things, but they can feel things when they interact with them.

Paul Rand: Okay. And who today is a candidate for this?

Nicholas Hatsopoulos: So in our clinical trial, we’re approved to implant subjects who have spinal cord injury that do not have function of their hand. Basically, they can’t make individual finger movements of their hands. And we’ve now expanded to include also amputees. So those are the two patient populations we can work with for our clinical trial.

Paul Rand: Okay. I’m curious, you see these examples where somebody will for the first time be able to hear or to see. What is the reaction early on when people are having this experience?

Nicholas Hatsopoulos: The one subject that really impressed me the most was actually an ALS patient. We did implant an ALS patient, and that’s a very devastating neurodegenerative disease. And he and his family, I think, were so moved by his ability to move a cursor when he couldn’t basically do anything, almost no movement. And that was really impressive and impressed the subject as well as his family and his child. I remember a video of his child just being so impressed that his father could actually move something.

Paul Rand: What is the biggest misconception that people have about understanding how this all works?

What part of the brain controls movement?

Nicholas Hatsopoulos: That it involves large parts of the brain. It’s not just localized in one part of the brain. That’s where we stick our electrodes in one particular area. But in the real world, when you actually move your real limb, it involves neurons throughout the brain. It’s highly distributed across the brain. And it also involves the spinal cord. The spinal cord has neurons as well, and they contribute to movement. And they also contribute to non-voluntary reflexive movements. So there are a lot of movements you make, you’re not even aware of.

Paul Rand: Right.

Nicholas Hatsopoulos: So I would say what people are maybe not aware of is that movement involves a huge part of the nervous system.

Paul Rand: So as you were talking about it, you’ve got electrodes that are picking up neurons firing, and how do you get from that to knowing that someone wants to reach right?

Nicholas Hatsopoulos: The way we build these, we have to build a decoder. Basically, these are electrical signals from many neurons, and we have to understand what do they mean? We have to understand the language of the brain. So how do we do it? It turns out, based on work that I had done in my lab with non-human primates, when these animals actually watched a cursor move as if they were moving it, but just not moving their arm, just watching it, those cells in motor cortex would activate as well. And so with that knowledge, we use it in humans. So what we do is we show them on the screen a robot arm or some avatar moving and we say, imagine you’re moving that.

Paul Rand: Okay.

Nicholas Hatsopoulos: Imagine you’re moving this virtual arm. And when it moves to the left, we record the brain signals and we then understand what those brain signals are when he’s imagining moving to the left or moving to the right. Once he makes all these imagined movements, when he’s watching things moving on the screen, we then build a decoder, which is basically a mapping between brain activity and movement.

Paul Rand: Okay. What are you seeing differently in your example you just gave of reaching left or reaching right? Are you seeing different parts of the brain firing?

Nicholas Hatsopoulos: Not so much different parts of the brain, but different cells are active.

Paul Rand: Okay.

Nicholas Hatsopoulos: So some fire when you move to the left, but not when you move to the right. Other cells fire when you move to the right, but not to the left.

Paul Rand: You have said that a flattened human cortex would cover about two and a half square feet. I guess it’s about four sheets of paper. Why does that layout matter?

Nicholas Hatsopoulos: Well, it turns out that what distinguishes us from other animals that have a cortex, but maybe don’t have the same capabilities and intelligence we have, depends primarily not on the thickness of the brain, but on the spatial extent, like how many sheets of paper, if you flattened it. So a mouse cortex, the cortex is the part of the brain that people think is involved in higher level thinking. And a mouse cortex, if you flattened it, is like a postage stamp size, but its thickness is not that different from us. It’s actually not a flat piece of paper. It has a thickness to it. Its thickness is not that different. It is a little different, but what’s really different is how big it is. We’re four pieces of paper. A monkey is maybe like one piece of paper, and that’s what distinguishes different animals and their intelligence is the spatial size of that.

Paul Rand: Let me go down and talk a little bit kind of on the neural tide element of this. So just before movement starts, there’s a wave that kind of sweeps across the motor cortext. What is that wave doing? What are you seeing when that wave is happening?

Nicholas Hatsopoulos: You can think of it like a wave in the ocean. It’s like if you look at one spot on the ocean, it’s like an oscillator. What we found was the amplitude, the magnitude of that up and down goes down, shrinks at movement onset. So it’s starting very high when you’re preparing to move and then it’s amplitude, the little wiggles shrink in size when you move. Now that’s in one spot in the brain, but if you look at many spots in the motor cortex, it turns out each little wave oscillation shuts down or shrinks at a different time and it forms a propagating pattern. So some sites shut down first, then other sites shut down later and it forms a pattern, a propagating pattern like a wave. And the analogy I make is it’s like for those of the listeners who remember driving a manual transmission car, you had a clutch.

So you had to put your foot down on the clutch to engage a gear. So when you put your foot down on the clutch, the engine is running, but you’re not moving. Okay? So I think of this as a whole series of multiple clutches in the brain and you’re engaging these clutches in a wave-like pattern to initiate the movement. When you’re preparing to move, all the clutches are down. So they’re basically the engine is revving -

Paul Rand: But not engaged.

Nicholas Hatsopoulos: But not engaged. Then you release the clutches to engage the engine.

Paul Rand: And what is releasing the clutches?

Nicholas Hatsopoulos: That we don’t know. We don’t know exactly how that happens. We have some modeling work where we modeled a network of neurons and it looks like if you provide input to this network and you set it up the right way, you can see that same pattern. So there’s an input, there’s like a go signal coming from somewhere in the brain. We don’t know where it’s coming from. When you decide, okay, I want to move, there’s a go signal where I don’t know where it is. It comes into the motor cortex and starts engaging those clutches.

Paul Rand: The wave isn’t just correlated with movement, it’s necessary for movement. How did you prove that it’s actually necessary to happen?

Nicholas Hatsopoulos: Well, we did some electrical stimulation experiments where we could electrically stimulate those electrodes with a pattern, with a spatial pattern that either mimicked the natural pattern that we see when you initiate a movement or a pattern that went in the opposite direction. So imagine you’ve got this cortical sheet, it’s like a piece of paper. The clutches are engaging from left to right to initiate a movement. Okay. So now we electrically stimulated either from left to right or from right to left. And we found that when we stimulated right to left, the movements, the monkey, we did these experiments with non-human primates, their reaction time slowed. They delayed their movement. So we could affect when they moved, when they initiated the movement. So that was some evidence.

Paul Rand: And what did that tell you when you could see that?

Nicholas Hatsopoulos: That told us that maybe this wave-like pattern is a necessary component to initiating a movement. If you don’t set up the cortex like that in that spatially organized way, if you could disrupt that for somehow, we tried to disrupt it with electrical stimulation. There may be other ways to do that. You can’t initiate a movement. You can plan a movement, you can think about moving, but you won’t actually engage the engine.

Paul Rand: How do you combine brain signals then and muscle signals to make a robotic hand move like a real one?

Nicholas Hatsopoulos: That’s a very important question because our brains have evolved to control muscles, not motors. So for example, we have a robot in our lab that actuates, that moves with motors. Motors are very different than muscles. Why? Because muscles are like springs and they have springy like action. And so they generate forces that don’t only depend on how much signal they’re receiving, but also how stretched out they are. So muscles are just different things than motors. And so our brains have evolved to control muscles. And so how could we do this? How could we actually activate muscles? There are at least two different ways to do it. One is, and there’s been some work done, a colleague of mine at Northwestern has done this, where you record from the brain, you decode the signals. In other words, you try to decipher what the animal is thinking of doing, and then you electrically stimulate the muscles from the decoder.

So if the decoder says move left, then you electrically stimulate the muscles that would move your arm to the left. And that actually would work with people who are paralyzed because their muscles are not the problem. If it’s spinal cord injury, it’s the spinal cord that’s been cut or been injured. Their muscles are fine, so we could stimulate the muscles. So that’s one way to do it. Another way would be to use a robot that has muscle-like actuators.

Paul Rand: And what would that look like?

Nicholas Hatsopoulos: Well, there’s some robots out there that have developed spring-like actuators that act, and it would give you more of the fluid-like motion we see in healthy individuals who move day-to-day. Part of that fluidity is due to the springy-like nature of the muscles. So another way, a third way to do it is create a computational model of the muscles. So you send the decoder to the model that creates forces like the muscles, and then the output of that model would go to a robot, a standard robot. And that’s actually something that we’re doing right now in the lab with my collaborator, Lee Miller at Northwestern.

Paul Rand: And so there are commands, reading commands are going out, but then of course, now you’re also getting sensation back in.

Nicholas Hatsopoulos: So we’re getting that feedback, touch feedback. We can also get proprioceptive feedback that goes to the somatosensory cortex, but ultimately to the motor cortex. And one of the challenges we’re facing today in the lab is that when we electrically stimulate the somatosensory cortex to simulate touch or to give the sensation of touch, that stimulation in turn disrupts motor cortex. And so what we’re facing with the problem now is that we’re trying to decode signals to move from the motor cortex, but then we’re receiving effects from stimulating somatosensory cortex and that disrupts motor cortex and it’s causing us problems. And so that’s a challenge we’re facing right now.

Paul Rand: Are they getting tangled?

Nicholas Hatsopoulos: Yeah, they’re getting mixed up. Right, exactly. But we somehow have to incorporate the fact that in the real world, there is this ongoing sensory feedback that affects motor cortex. So we have to incorporate that into our decoders so that it doesn’t disrupt what the subject wants to do in terms movement wise. He wants to move to the left. Okay, but he’s touching something. We don’t want that touch signal to disrupt his intention to move to the left, but that’s a challenge we’re facing now.

Paul Rand: Have you ever wondered who you are, but didn’t know who to ask? Well then join Professor Eric Oliver as he poses the nine most essential questions for knowing yourself to some of humanity’s wisest and most interesting people. Nine Questions with Eric Oliver, part of the University of Chicago Podcast Network. 

Paul Rand: Your lab has shown that feedback does improve how well somebody is controlling your device. Talk to us about that feedback and this whole idea of kind of closing this loop.

Nicholas Hatsopoulos: If I want to grab a cup of coffee, so I have this cup right here in front of me. Once I grab it, I have no sense of how much force I’m applying on it. So I don’t want to crush the cup if it’s a paper cup, or if it was an egg, for example, I don’t want to crush the egg, or I don’t want it to drop, so I don’t want to apply too little force. How do I get that information? I don’t get it from my eyes. I get it from touch. So I know how much force I’m applying. And so that’s what these subjects in principle can do. They can feel what they’re grabbing or interacting with and adjust what motor output they’re going to generate so that they grab it properly without dropping or crushing it.

Paul Rand: Okay. There was a time, of course, that this was considered one-way traffic, the brain out to the limb. When was it that that changed and they realized it was coming back?

Nicholas Hatsopoulos: Oh, no, that’s been known for -

Paul Rand: A long time?

Nicholas Hatsopoulos: A long time. In neuroscience proper. In the field of brain computer interfaces, it’s new because it really started, I think, out of Pittsburgh, University of Pittsburgh. I mean, so up till that time, people were just ignoring that kind of feedback. They relied solely on vision, which is important. Vision is an important sensory modality. But actually, there were other researchers along the way that were thinking about it, but generally in the field of brain computer interfaces, it’s new, maybe 15 years old, 10, 15 years old. But in neuroscience proper, the field broadly, it’s been known for a long time, over a hundred years.

Paul Rand: You’ve got a paper that’s recently out, is that right?

Nicholas Hatsopoulos: Yes.

Paul Rand: And you talked about the difference between imagined and real movements coming down, I think, to the state of the limb. Unpack that a little bit for me, if you would.

Nicholas Hatsopoulos: Okay. So we took advantage of a unique opportunity with our human subjects that we have now implanted in Chicago, because they’re not completely paralyzed. So in fact, they can move their affected arm, so they can reach for things, but they can’t grab things and manipulate their fingers. So what we said, “Okay, let’s take advantage of that and learn something scientifically.” So we said to the subjects, “Either imagine making a movement without moving or reach to a target by moving your arm, or we will passively move your arm to the target.” So there were three conditions, imagination, active movement, and passive. And the scientific question was, “Is the brain acting the same way? Is the representation of movement the same under those three conditions?” 

And the short answer is that no, it’s not the same. And in fact, active movement and passive movement look much more similar than imagine movement. So this is interesting to me scientifically because it suggests that when we train, at least when we train these decoders, we asked the subject to make imagine movements. So the decoders may not be the natural way the brain works if you’re really moving. So the brain computer interfaces we use are fundamentally different than the way the natural system works. I didn’t expect this, that active and passive movements would be more similar to each other than imagine movements. Because I would have thought that maybe imagine when you think about moving and actually move your arm actively, it would be very similar, but they’re not so similar. So I guess what’s the practical implication maybe we need to incorporate, like one idea is exoskeletal robots, robots that actually passively move the limb in subjects that have residual proprioception. We could actually passively move the limb and allow them better control by passively moving because we found that passive movement is more similar to active movement. I don’t know. So it’s one idea.

Paul Rand: Yeah. And as you think through even rehabilitation and how that’s working, do you look out and say, “What we’re looking at now, I see an impact in five years or 10 years.” And what kind of an impact do you think these types of insights are helping gain?

Nicholas Hatsopoulos: So what we’re doing primarily would be considered more assistive devices. These are devices that help subjects interact with the world. But you’re talking about rehab. Rehab is something a little different, which means the way I understand it is that we’re actually trying to take advantage of plasticity in the nervous system to improve the natural movements of the subjects. So for example, if you have a stroke, there are technologies and ideas from brain computer interface work that could be used to rehabilitate somebody who has a weakness or no movement in their limbs or very little movement to enable them to move their arm or hand better and take advantage of neuroplasticity by using electrical stimulation. Yes, there’s some thoughts about. In fact, there are experiments that have been done primarily in animal models that have shown that you can induce plasticity for rehab purposes.

Paul Rand: And as you think about what is the optimistic outcome of this type of work, how would you describe it?

Nicholas Hatsopoulos: Be something that people with spinal cord injury, with amputation, with ALS, with certain kinds of stroke could incorporate this system, not just in the lab, which is what we currently do, but in their homes. So they could actually, it could help them with daily living. It could help them dress themselves or feed themselves or whatever they’re doing. And it would just improve the quality of their lives at home or in the world outside of the lab. So I think there’s promise.

Paul Rand: As you think about the biggest questions that still remain and what excites you most about the work that you’re able to continue doing, where does that take you?

Nicholas Hatsopoulos: So open questions. Well, one of the issues is the implant itself. You want an implant that’s reliable, that can record from the brain for a long amount of time. That’s still an issue. It’s still a challenge. And it’s really a bioengineering problem. Finding some interface with the brain that’s reliable for long periods of time. So that’s one challenge we’re facing. Another would be robotics, improving end effectors, things that you can control that resemble the natural arm and hand. So improvements in robotics is something, and just devices that you could control that would be more natural ways of interacting. And then yeah, I think those are some of the big issues. And also recording from, also recording from more brain areas.

Paul Rand: So there’s been extraordinary progress and then there’s some extraordinary possibilities and opportunities that are still in front of us directly and longer term.

Nicholas Hatsopoulos: Yes, there are.

Paul Rand: You started by talking about your interest of a robot being able to do Greek dance. How close are we to that?

Nicholas Hatsopoulos: Well, there are actually impressive developments in robotics now and I’ve seen autonomous robots doing incredible moves. We’re almost there. I think we might even be there. Maybe not Greek dances yet, but we’re close.

Paul Rand: Bold and curious. These words describe the guests who you hear on Big Brains who are shaping the future of research. The University of Chicago is launching a new campaign called Chicago Minds to further advance this kind of groundbreaking work and expand the university’s global impact.

Lea Ceasrine: Big Brains is a production of the University of Chicago Podcast Network. We’re sponsored by the Graham School. Are you a lifelong learner with an insatiable curiosity? Access more than 50 open enrollment courses every quarter. Learn more at graham.uchicago.edu/bigbrains. And if you like what you heard in our podcast, please leave us a rating and review. The show is hosted by Paul M. Rand and produced by me, with help by Eric Fey. Thanks for listening.