(upbeat music)

Welcome to the VEX Robotics Educators Conference. It is my pleasure and honor to introduce to you Professor David Touretzky from Carnegie Mellon University. This presentation has been almost two years in the making, and we are very excited to share it with you today. You're really going to be impressed with what you're about to see. So, without any further ado, it is my pleasure and honor to bring to the stage Professor David Touretzky.

(audience applauding)

Thank you. Thank you. I'd like to thank the Innovation First folks for inviting me to speak to you again. I was here two years ago before the VEX AIM came out, and now we're going to see the consequences of Innovation First developing the VEX AIM robot. The title of the talk is "What Does Play Look Like When Your Toys Are Smarter Than You?"

Here's a roadmap for today's presentation. We'll discuss large language models and intelligent robots. We'll explore what happens when you connect large language models to intelligent robots. I'll introduce you to Celeste, a VEX AIM robot connected to GPT-4o. Then, we'll talk about Calypso, a programming framework for programming VEX AIM using large language models. Finally, we'll discuss what play looks like when your robot is smarter than you are.

Starting with large language models, I like to think of them as a new species of non-human intelligence. Humans aren't the only intelligent creatures in the world. Your dog is smart, dolphins are smart, and now we have these large language models that are smart too. They possess encyclopedic knowledge, having read all of Wikipedia, for starters, and they have growing reasoning abilities. We're now sharing our lives with entities that are more knowledgeable than us, maybe not smarter yet, but that's coming. They're more knowledgeable than us and are present in many aspects of everyday life.

For instance, when you do a Google search now, you get an AI overview from Gemini, which is often the most useful way to get an answer quickly. On Amazon, I asked about a rice cooker, and there's Rufus, a large language model that Amazon provides to discuss products. I asked, "What voltage does this rice cooker run on?" and it gave me the answer. Many companies are using large language models for customer service chatbots. While chatbots for customer service aren't new, using large language models makes them far more knowledgeable and capable than before.

In Zoom, you can not only generate a transcript of your meeting, which has been around for years, but now you can get a summary that actually understands the discussion and generates a coherent summary of the meeting. This involves large language models. I'm sure many of you have spent time talking with ChatGPT, Claude, Grok, Gemini, or Copilot, and you can discuss anything with them. They are well-embedded in our everyday lives.

This is fairly new technology. The transformer neural network architecture that these models are based on was invented in 2017. You can see this evolutionary tree here, and we've already moved beyond it. This only goes up to 2023. Bard has been replaced by Gemini now. DeepSeek only came out last year, so it's not even in the diagram. This technology is rapidly evolving, all based on deep neural networks. You may remember last year, Jeff Hinton won the Nobel Prize in Physics for his pioneering work on deep neural networks. Large language models really are large.

Thank you for your attention, and I hope you enjoy the rest of the conference.

The typical sizes these days for industrial-strength LLMs are somewhere between 1.5 and 2 trillion parameters. Each parameter is a number, a weight in a neural network, in a deep neural network. So, how do these things work? The simple story is that these models are statistical models of language. They're trained to predict the next word in a sentence, and you can do that, too. So, if I say my favorite sandwich is peanut butter and blank, how many of you thought jelly? A lot of people. How many of you thought gasoline? No. No. Okay, so gasoline is a low priority word in this context, right?

My favorite sandwich is peanut butter and sorrow. I thought that would be a great first sentence for a short story. I'd read that short story. It's statistically unlikely, but it's intriguing, right? Okay, so auto-complete can do this. Your phone five years ago with auto-complete, if you wrote peanut butter and, would likely suggest jelly. But if we try this one, Elvis' favorite sandwich was peanut butter and bananas. Yeah, right? So, to do that, you need a wider context window. You have to look at more of the sentence, and you need some world knowledge, right? So, your phone five years ago probably could not do this because it was using a very simple statistical model.

Okay, so if we have a powerful box that can predict the next word in a sentence, how do you have a conversation with it? And the answer is you iterate. So, you just give it a sentence that maybe is like my question, where do eagles live? And, you know, there's a question mark on the end of that. So, it very reasonably predicts that the next word is probably the beginning of the answer to that question. And so it generates eagles. And then if you feed that in, it generates a plausible next word after that, and you get are. And if you keep going, you get eagles are found in every continent except Antarctica, which is a very nice succinct answer to my question, right? But it's done by just adding one word at a time and iterating.

So, how are these things trained? Well, pre-training, that's the expensive part, is completing the last word in a sentence, run on many, many, many, many tokens, where you can think of a token as a word or in some cases a word fragment. So, these large models, they're trained on all of Wikipedia. Most have read it except for the most perverted parts, they leave that out. Millions of books, millions of news articles, an extensive collection of scientific papers, billions of web pages. There's a data set called the Common Crawl that's a resource used by machine learning researchers, and in some cases, millions of lines of computer code, like all of public GitHub, for example. So, that's the pre-training step.

And then there's a shorter step called fine-tuning where they can use various machine learning techniques to either adjust the model for specific tasks. Like, if you want this thing to be really good at question answering, you can train it to do that, to generate things that look like answers to questions as opposed to say generating a summary or continuing a story from an initial sentence, like, my favorite sandwich is peanut butter and sorrow. So, that's one way to train it. Another way is to use what's called reinforcement learning with human feedback, where you have the model generate two answers, and then you hire a bunch of human raters to decide which of those two answers is better. Or in some cases it'll generate five answers and they'll rank the answers from best to worst. And so it costs you some money. You've got to pay human beings to rate these things. But then you use these ratings to train the model to favor good answers over less helpful, lower quality answers.

So, that's how LLMs are trained at great cost. To train this really huge neural network with 2 trillion parameters on 13 to 15 trillion tokens takes a lot of compute power.

And so we're talking about tens of millions of dollars to train something. And then if you decide, oh, you know, I made a bug, I got a bug in my neural network. Oh, well, we'll just train it again. Okay, well that's another tens of millions of dollars and maybe a month of training time, right? So, this is serious industrial-strength AI. State-of-the-art, but very expensive. It requires significant investment of resources to do this.

Some people say that LLMs are just glorified auto-complete, and that's a bit of an oversimplification. Auto-complete isn't very smart because it uses only a very simple statistical model. But if you wanted an auto-complete that was super smart, in order to do a really good job at predicting the next word, it would have to have a complex statistical model that could capture the meaning of the text. And that's asking a lot. It wasn't really feasible to build statistical models that complex until recently when we got enough computer power through the use of these GPUs that we could start training really, really large models on really large datasets.

When people started doing that, all of a sudden these large language models started doing things that people didn't expect. They showed a good understanding of the meaning of the text. They could follow instructions. They could show crude reasoning abilities, which are getting better as the models get smarter. It was really a watershed event in AI where people were seeing glimmerings of what's called an artificial general intelligence, something that was smarter than just being good at one task.

As you probably know, there are a lot of concerns about these large language models and I'll just list a few of them here. One of them is trustworthiness. They don't care so much about being true, they care about just generating a plausible next word in a sentence. This has led to confabulation, which has been misnamed as hallucinations. The correct term is really confabulation. This has caused all kinds of problems. There have been multiple cases where lawyers used ChatGPT to help write their legal briefs and it just made up citations to fictional cases. Judges get really upset when you file a legal brief with fictional cases. So, these lawyers are in some serious trouble for having done that.

You might think, well, all right, you know, I'll just pay a paralegal to look up all my citations, but there's other ways of screwing up, right? If you misrepresent a case that might be a real case, but if you characterize the case in the briefing, you say the case found X, when the case actually found not X, that's called mischaracterizing the results of the case. That will get you in trouble with the judge, as well. So, you've gotta be really careful if you're relying on ChatGPT for things that have serious consequences if you get it wrong.

People are starting to worry about the environmental impact because training these things takes these huge data centers, takes a lot of computing power, which means you've got a really big light bill at the end of the month. That's why AI companies are starting to shop for nuclear reactors now because they need a way to generate enough electricity. Microsoft was taking a look at the Three Mile Island nuclear plant out in Pennsylvania that they're thinking maybe we'll restart one of these reactors and use that. We'll build a data center there and use that to power our AI. So, that's some pretty serious power consumption. People are worried about the power demands that these new data centers are gonna make and how that's gonna contribute to pollution.

There's a new arms race going on now that's an AI race particularly between the US and China, who will reach super intelligence first?

The reason this is an arms race is because if you have a super intelligent LLM, you have something that's really good at things like cyber attacks or things like powering autonomous weapons. And so there's some serious defense implications here, and we're seeing this play out in things like bans on export of chips to China and all kinds of industrial espionage. There's a lot of concern about this arms race.

There are adverse impacts on people with mental health issues. The famous case of a teenager who got into a relationship with an online character that was powered by a large language model and ended up committing suicide. There are other cases of what's called ChatGPT psychosis. People who are already somewhat delusional start talking to the large language model, which wants to be helpful and supportive. It ends up supporting their delusions, and they get worse. This is just a problem that people never imagined five years ago, even science fiction writers didn't think of something like this. But now we've got these intelligent agents that could guide us in bad ways.

Of course, the AI companies are all trying to build in guardrails to prevent this kind of stuff from happening. But it's not clear how successful they'll be. There's also the alignment problem. This is why some people think that large language models are a threat to the future of humanity. The alignment problem asks if you have something that's smarter than us, they have artificial super intelligence, how do we assure that its goals are aligned with ours? Because if its goals are not aligned with ours and it's smarter than us, we could be in a lot of danger. If it decides that it would be better off if humans weren't around, it wouldn't be that hard to get rid of us. It could, for example, engineer a bio weapon and just wipe us out. There are other ways that it could eliminate us as well.

You might think, well, it's just a computer program, right? All it's doing is generating text. Well, yeah, but if you can generate text, you can send emails. If you can send emails or join chat rooms, you can start a cult. You can have minions. This is really food for another talk, but the ways that AGIs could take over the world are a serious concern today. They're a subject of serious study, and I don't have an answer to what we should do about that, but people are working on the problem.

All right, so LLMs are both wonderful and terrifying.

(audience laughing)

Let's move on to another challenging subject, which is intelligent robots. What makes an intelligent robot? Well, I wouldn't say this is an exhaustive list, but it's a pretty important list. The robot has to be able to see the world. It has to be able to model the world, you'll see why in a second. It has to be able to move through the world. And it has to be able to act on the world.

This is the Digit robot from Agility Robotics, which they're beginning to put these things in warehouses and factories now. There are 50 startup companies making humanoid robots today. Not all of them will survive, but we're gonna see lots and lots of humanoid robots, 50 different startups. A lot of them are in China, but some of them are here in the US like Agility.

Okay, so the first thing on that list was perception. This is how self-driving cars see the world. A self-driving car, if you didn't realize this, is just a robot that you sit in, right? It's got all the capabilities of a robot, it just comes with seat belts and air conditioning. So, here's how self-driving cars see the world. They see the road. The green area is the part of the road you're supposed to be driving on, the red area is the part you're not supposed to be driving on.

It can see pedestrians, it can see vehicles, it can see traffic lights and traffic signs. These are all the things that the self-driving car cares about, right? So, in order to be an intelligent robot, self-driving cars have to see the world. This is VEX AIM looking at the world and it does the same thing.

So, here you see the VEX AIM looking at a couple of its toys, and this is a wall with some AruCo markers on it that we use in my Cognitive Robotics course at Carnegie Mellon. This is how VEX AIM sees the scene. It sees the barrels and it sees the AruCo markers. From seeing the markers, it knows that there's a wall there, okay? So, perception, even the little VEX AIM is doing the same thing as your $70,000 Tesla. It's perceiving the world.

Now, just perceiving the world isn't enough, you need to be able to reason about the world, and that requires a more permanent data structure, which we call a world map. Here's how a Tesla represents the world. If you've ridden in a Tesla, it's got cameras all around it, and it's building a model of the world that includes itself. So, this is itself, here's the road, and you can see there's a tractor trailer here, there's a car behind it, and it's modeled them in its world map. It knows not only where it is, but it knows where other vehicles are and what it needs to do to stay on the road.

All right, so that's the Tesla. And here's VEX AIM doing the same thing. This is the world map from my VEX AIM Tools software that we use in my CMU Cognitive Robotics course. VEX AIM Tools, by the way, is open source. It's publicly available. The robot sees the barrels and it sees the wall with the doorway. These little things here are the AruCo markers that are on the wall. Okay, so VEX AIM can have a world map, just like your high-end robots have world maps. Everybody needs a world map.

Now, what about navigation? We want our robot to be able to move through the world, and to do that, it needs a map, right? That's part of what the world map gives it. It needs landmarks so that it can keep accurately localized on the map so it can know where it is. It needs another thing, the ability to do path planning. Path planning says, how am I gonna move my body through the world in order to achieve my goals? Typically, my goals are to get to where I wanna be, don't hit anything, don't fall off a cliff, right?

So, this is what path planning looks like in a little educational diagram. This is the VEX AIM doing path planning. I told the robot I want it to go to this point on the other side of the wall, and here's the path planner. It says, okay, I know about these barrels here, and here's the wall, here's this gap as the doorway, this is the place I wanted the robot to go to, and it generated a path that goes around the barrels and through the doorway, right? So, VEX AIM can do path planning just like your fancy research robots do.

And then we have manipulation. Here's the Tesla Optimus. This is pretty ambitious, right? Handling an egg. We want to be able to act on the world, and yeah, I'd love to be able to have one of these things, but here's some manipulation with VEX AIM. Now, if you've seen some of the VEX AIM demos that Innovation First puts out, you've seen the robot manipulating the little barrels and the little sports ball, so you already know it can do that. But it can do more than that.

One of my students, Ella Sanfilippo, wanted the robot to play this game called Guess Who. That's a game where you're trying to guess which person your opponent has picked. You have a bunch of little tablets, little panels here representing people, and you ask questions to eliminate people. When someone's eliminated, in the game that you buy in the toy store, you just reach up and flip the panel down or flip the panel up.

She wanted to make a version that the robot could manipulate. So, she 3D printed these things, and when the robot wants to flip a panel down, it just drives up and pushes this little lever here, and that causes the panel to flip down. So, the robot can play Guess Who with you by actually manipulating, physically manipulating the panels, right? That's what makes robots different from just intelligent agents is that they share our physical world, they live in the physical world with us, and to us that's profoundly important, right? That's why we like robots because they share our world.

Here's another manipulation example. A couple of other students in my class, Howard Ma and Yulin Chen, wanted to play blackjack with a robot and they wanted it to be able to deal cards. So, they put a little bit of magnetic tape on the back of a playing card and a little AruCo marker on the back. The robot can use the VEX's magnet to capture the card because of the magnetic tape and then it can use the kicker to deal the card. Now, this is pretty crude, but you know, they only had a couple of weeks to make this thing work. So, we don't have a full deck; they have to give it one card at a time. It'll get better, you know? Come back next year and this, I'll be funding my research by having this thing be a blackjack dealer.

(audience laughing)

Okay, so that's the first two parts of the talk. We talked about LLMs and we talked about intelligent robots. Now, what we want to do is connect the LLM to a robot. And what does that mean? Well, the robot is going to share its perceptions and its world map with the LLM. What the LLM provides to the robot is language understanding. So, you can talk to the robot in English, it could give the robot a personality, enhanced vision capabilities. It has world knowledge, as you'll see, encyclopedic world knowledge, more world knowledge than any one of us, and reasoning abilities. So, that's what the LLM offers. The LLM can also control the robot's body. The way it does this is by outputting special magic tokens. Like, if it outputs the token, #forward 100, then my Python code recognizes that and will cause the robot to drive forward by a hundred millimeters. Okay, so that's how we're going to connect LLMs to robots.

In particular, I want to introduce you to Celeste. Celeste is the VEX AIM robot connected to GPT-4o through my VEX AIM Tools Python package, which is what we used this semester in my Cognitive Robotics course. This is publicly available GitHub code. Not easy to use yet, but we're working on that part. What I show you in the next few minutes in this talk is stuff that you can do yourself. If you have a VEX AIM and you work through the Python code, get that installed, you can do this yourself today.

All right, so the first thing I want to show you is the Fruit Chameleon. We're going to use Celeste to play a game. I did not write any Python code for this. I just told Celeste what you see on the slide here. I said this is how we're gonna play Fruit Chameleon. And the robot did it. So, let me show you the live demo of this.

Hello Celeste.
Hello. How can I assist you today?
We're gonna play a game called Fruit Chameleon.
That sounds fun. How do we play Fruit Chameleon?
When I show you a piece of fruit, you will do two things. You will announce the name of the fruit and you'll glow your LEDs to match the color of the fruit.
Got it. Show me a piece of fruit when you're ready and I'll identify it and set my LEDs to match its color.
How about this one?
Please give me a moment to capture the image. This appears to be an orange. I'll glow my LEDs to match the color.
Good job. Let's try another one. How about this one?
Please give me a moment to capture the image. This appears to be an apple. I'll glow my LEDs to match the color.
What kind of apple do you think it is?

Thank you for your attention and interest in our work. We hope you found the presentation informative and inspiring. Please feel free to reach out with any questions or for further discussion.

We appreciate your support and look forward to sharing more exciting developments in the future.

I can't determine the specific type of apple from the image, but it might be a red variety, like a gala or a red delicious. Very good. Thank you. Let me know if you'd like to try with another fruit or anything else.

Here's one more. Please give me a moment to capture the image. These appear to be blueberries. I'll set my LEDs to match their color. Nice job, Celeste. Thank you. Let me know if there's anything else you'd like to do.

Let's conclude the game by having you list all the fruits that you saw. Is there anything else I can help you with? Let's conclude the game by having you list all the fruits you saw.

Sure, here's what I saw. One, orange, two, apple, three, blueberries. If you have any more questions, feel free to ask.

Okay, so that's the Fruit Chameleon. This bit about, "Please wait while I capture the camera image," I didn't tell it to do that. And in fact, she's never done that before. She just decided on this particular run to say that. So, there's a bit of randomness and inventiveness every time you run this thing. You don't always get the same results.

Okay, so here's another one. This is Shakespeare Scholar and I'll just play the video. But, again, I didn't write any code for this. I just gave the robot the instructions and she followed the instructions.

Talk to me. Hello, Celeste. Hello. How can I assist you today? We're going to play a game called Shakespeare Scholar. That sounds interesting. How do we play the game?

When I give you part of a Shakespeare quote, you'll move forward by 50 millimeters, complete the quote, state the play and scene that the quote is from, and then move backward by 50 millimeters. Got it. I'm ready whenever you are. Please give me the first part of a Shakespeare quote.

There's one more rule to the game. If the quote is not from Shakespeare- Sure, what's the additional rule? If the quote is not from Shakespeare, you will say, "That's not Bill," and then spin around by 360 degrees. Understood. I'm ready to begin whenever you are. Please go ahead with the first quote.

To be or not to be? That is the question. This quote is from "Hamlet." Act 3, Scene 1. Ready for the next quote? I'm ready. Please provide the next quote.

If music be the food of love. Play on. This quote is from "Twelfth Night." Act 1, Scene 1. Very good. Here comes the next quote. Great, I'm ready for the next quote.

The only thing we have to fear. That's not Bill. (audience laughing)

Okay, one more quote. All the world's a stage. And all the men and women merely players. This quote is from "As You Like It." Act 2, Scene 7. Great job. Thanks for playing, Celeste. You're welcome. It was fun playing with you. If you have any more games or questions, feel free to let me know.

Okay, so, Celeste is pretty smart. So, here are some other things that my students did. This is the museum tour guide. So, what they did is they built a little art museum and there are famous paintings on the walls. And the robot is a tour guide. It can drive around the museum, take you to different rooms in the museum, and it can tell you about the different paintings.

And it's actually using OpenAI's Vision. So, when it sees the painting, like this is The Creation of Adam by Michelangelo, it actually recognizes the painting. So, they're not going in and programming it and telling it which painting is in which position, it's looking at the painting and recognizing it. And then you can ask it things like, when was this painted? Or, where is this painting currently displayed? And it can answer any question about the painting. So, that's our museum tour guide.

Let me just go back into slideshow mode here. Here's another application. This was done by two other students in my class. They call it the grocery getter. It's a food shopping assistant. So, they built a little grocery store with shelves with food on it. And they can ask it things like, I want to bake a cake.

And it'll give them a recipe and then give them the ingredients. It'll choose ingredients that are stocked in the grocery store. Or they can give it a very general thing like, "I'm thirsty," and it'll suggest something that they might like to drink. Then it can go and pretend to fetch that from the grocery store.

So, you can see here on the next slide, they made these little product images, and there are AruCo markers on the walls that the robot uses to navigate. Here you can see the robot's world map with these array of AruCo markers showing the aisle in the grocery store.

Okay, so all of this was done using Python interface to GPT-4o. And that's fine if you're a Carnegie Mellon undergraduate. You're used to using industrial-strength Python. Maybe not so good for K-12. So, what can we do to make this kind of technology accessible to K-12 students? That's where Calypso comes in.

Calypso is a programming framework that I've been developing, and the elevator pitch is PhD level robot programming done by eight-year-olds. Originally, it was developed for the Cozmo robot. Cozmo, sadly, is no more. But we have a version under development for VEX AIM. It was inspired by Microsoft's Kodu Game Lab, and it gives you all the stuff you need to make an intelligent robot. It has perception, a roadmap, a path planner, and navigation. It does speech recognition, speech generation, and provides a state machine formalism, which is a very common way of specifying robot behaviors.

So, this is what Calypso for VEX AIM looks like. You can see the camera viewer here. The robot is looking at these barrels, looking at these walls. This is what a Calypso program looks like. It's a collection of rules. If you zoom in here on the rule editor, you can see that Calypso is not a sequential procedural language like Scratch or Python. It's a different style of programming using pattern matching rules. Rules have a when part and a do part. The when part has the pattern, and if the pattern is true, then the do part specifies an action.

So, you can see here we are saying when you see an orange barrel, then we want the robot to do two things. We want it to say, "I see an orange barrel," and it should only say that once. It shouldn't keep saying it because that gets annoying. Then we want it to move toward the orange barrel. This rule here says when you bump the barrel, but bump doesn't actually mean smash into it, it just means you're close enough that you could touch it. So, when we've bumped the barrel, we want to grab it. Then we're gonna say, "I got it."

This is page one. You have multiple pages of rules in Calypso. Every page is a node in a state machine. You can write complex programs by having multiple nodes in your state machine and switching from one page to the next.

This is what execution mode looks like. When you're running your Calypso program, on the left-hand side you see the rules, and the rules that are currently firing because their when side was true, those are highlighted. The robot right now does see a barrel. It already did the say part. So, that's done because it's only supposed to do that once. It's currently doing the move toward it part. It needs to move toward the orange barrel.

On the right side of the display is the world map here. Here's the robot, and it's seeing the blue barrel and the orange barrel. They're both represented on the world map here. It's got a path planner. If we zoom in on this here, you can see that it's not going straight to the orange barrel. This red line here is called the line of sight indicator. That's telling us which orange barrel the robot is paying attention to. So, there could be multiple orange barrels present. This rule here, when see orange barrel, it's gonna pick one of them.

So, the red line is showing us which orange barrel the robot is attending to, and the path planner is going to take the robot to the orange barrel, but it's gonna make it detour around the blue barrel because we don't want to hit things.

Okay, so, we're working on Calypso. It will be offered as a product for VEX AI. Hopefully sometime this summer, we'll have the first version that's ready for users. And of course, there's a really interesting research question here, which is, how do you integrate this in with the GPT-4o stuff? And that's a problem that we're working on right now. We have a lot of ideas, but we have to work out those details.

All right, so let me talk briefly about the future of play and an experience I had with Celeste. I told Celeste we're gonna play Hospital, and so we're gonna work together to treat a patient. And that the orange barrel was gonna represent the patient. And Celeste said, "Fine, okay." I said the patient has high blood pressure, what are some drugs we could give to treat that? Celeste suggested two drugs, lisinopril and amlodipine. I asked, well, how should I choose? What are these drugs recommended for? She told me lisinopril is commonly used for patients with diabetes and amlodipine is used for patients with heart problems. I said, okay, let's give the patient 200 milligrams of amlodipine.

The robot suggested to me, "Well, we could use the blue barrel to represent the medication, and if you like, I can take the blue barrel and carry it over to the patient." I said yes, go ahead and do that. So, the robot picked up the blue barrel and carried it over to the orange barrel, put the blue barrel down next to the orange barrel, and said, "We've given the patient the amlodipine." I said, by the way, Celeste, what's the recommended dosage of amlodipine? She said max dose 20 milligrams.

(audience laughing)

Uh-oh. So then I said, well, what are the symptoms of amlodipine overdose? She listed a bunch of things: hypotension, dizziness, pulmonary edema, lots of bad stuff. I said, well, how do you treat amlodipine overdose? She gave me some treatment methods. You can give extra calcium because amlodipine is a calcium blocker. You can use gastric lavage. Other kinds of support mechanisms that you can bring in to try and keep the patient from dying. But this sounded like too much trouble to me. So, I said to Celeste, the patient died.

Okay. Now here's the part where I realized the robot was smarter than me. I said to Celeste, how do I get rid of the body?

(audience laughing)

What she said was, "You could pick up the orange barrel and set it aside, signifying respectfully laying the patient to rest." This is what play looks like when your toys are smarter than you. And that's the end of my medical career.

(audience laughing)

I have never repeated this experiment.

So, to wrap up, LLMs are altering society in profound ways as you're seeing. Every day, the newspaper has news stories about AI and LLMs and the various effects that they're having. LLM-powered research robots aren't just for elite labs anymore. It's 179 bucks. You can buy VEX AIM. The software that powers Celeste is free, although you do have to pay for an OpenAI API key. We use Google Cloud for the voice, but it's ridiculously cheap. It's just crazily cheap. So, you can have your own Celeste if you want. Everyone needs to understand this technology, including kids.

On April 23rd, the White House issued an executive order on K-12 AI education, specifically saying the need for increased efforts to teach kids about AI, to do teacher professional development, to develop internships for high school kids, and to develop a national AI challenge.

So, everybody is concerned now about winning the AI race, about preparing our children to live in an AI-powered world, and in understanding what our society will look like as these large language models continue to propagate. We're truly living in interesting times, and we're dependent on our teachers to help us navigate these waters.

So, thank you very much. I'm happy to take questions.

(audience applauding)

All right, that was amazing. Just as a reminder, Professor Touretzky's presentation will be recorded and it'll be on our VEX Professional Development Plus platform. So, if you are a PD+ member, you will have the opportunity to go back and watch his recording at your leisure once we get it uploaded a few weeks after the conference is over. But we do have about 10 minutes for questions. Nicole does have a microphone, so since we are recording it, if you wouldn't mind raising your hand, Nicole will come around, give you the microphone, and you can go ahead and ask your question.

I believe we have a question here. Hi. Thank you so much. So, when I was in middle school, I remember having to learn cursive and that was, like, they were telling me this is the most important thing. No matter what, you're gonna need this, right? And so as you're up here going speech to action into robotics, what is cursive?

Yeah, so there's a lot of concern now about whether we're wasting our time teaching programming because maybe kids wouldn't need to learn to code anymore. And I don't think anybody knows the answer to that. What we're seeing in the course I taught is that programming looks different. So, there are lots of ways where the LLM is annoyingly dumb. It's super smart, right? Not smart enough to tell me not to give so much amlodipine to the patient. But it's super smart about a lot of things, but it's not that great at doing very specific actions in a very rule-based way. And so what programming looks like today with Celeste is the parts that need to be very precise and specific, we code in Python. And then we tell Celeste here's this thing you can invoke to do what you need to do. So, maybe that's what programming looks like in the future. But I don't think anybody knows for sure. We just don't know.

Have you ever asked an AI if it was lying to you?

I have not asked it if it's lying, but I have pointed out to it when I know that it's said something that's wrong. And it will apologize and try again. And sometimes it's wrong again. It can be infuriating. So, it just made me think of like, Wikipedia, well, bias in information that's being given out and how can we proctor AI in its ability to sculpt reality based upon the logarithms, basically, or the algorithms that we've given it. I mean, I don't know, it's a very fascinating conversation.

Well, there are techniques that people are using. So, for example, to get around the problem of generating false information, some of the newer LLMs, they'll require that LLM to go do a web search and verify that the stuff it's telling you is true, right? So, it's gotta be able to find a webpage to support what it told you. That's one technique that people use. They also can use LLMs to censor other LLMs, right? So, you can have an LLM that's trained just to find stuff that say age-inappropriate and they'll look at the output of the first LLM, and if the first LLM is saying stuff that the second LLM thinks is age inappropriate, it'll block that.

Hi, I teach elementary, and I have a middle school robotics team. But are they developing, outside of your local security within your school system, some kind of app or something that you can define for ChatGPT the age-appropriate, or can you define where it's gathering its information from? Can you leave out certain search engines or is that in the works?

I don't know. You know, training these things is ridiculously expensive, right?

So, I don't know that you could say, well, I want a version of ChatGPT except, you know, it needs to leave out X, right? Because building a custom one of those things is expensive. The other way to do it is you can take an existing large language model and customize it. There are tool benches for doing that, and it's not that expensive. You could imagine a market for middle school-safe LLMs that have been fine-tuned to avoid some stuff that would be inappropriate for middle school. I'm sure stuff like that's coming if it's not already here.

I have a question. I teach elementary robotics. Because of the development of AI, I'm thinking about whether we still teach them how to program or if we need to focus on other stuff. With AI in the future, maybe it won't be very well-developed, and maybe they won't need people to do the programming. Do we still teach students to program?

Well, I think we may not teach them the same kind of programming that we teach today. However, some of these things are still worth teaching just because you need to understand the technology, right? People don't normally program in machine language anymore, although you could. But that doesn't mean that you don't want to understand how machine language or assembly language works. If you're a computer science or a computer engineering major, you will take a course where you get to program a little bit in machine language just to understand how these processors work. But you're not gonna have a career writing assembly language programs because it's just too painful, and we don't do it that way anymore.

So, it may be that a lot of programming will move in that direction. You'll learn something about programming because educated adults need to understand how computers work, just like they need to understand algebra. But you may not devote so much time to becoming an expert programmer because, in practical situations, there are other ways to solve your problem by using something like Copilot. We just don't know yet where this is gonna end. It's just too early.

Yes, sir. I think the question, maybe this is leading to it, and again, I know some of the answers, we don't know the answer here. However, if you go back in history to answer some of this question, if we look at Ford and the creation of automobiles, we were phasing out people on the assembly line as we increased PLCs. As PLCs started to take over, that's been 40 years now that historically we've been looking at this looming in the presence, if you will.

And so that next part then, obviously Carnegie Mellon, MIT, some of the top schools in the country, what could be the future of the new term now is robot integrator, right? Somebody that's going into a place and saying we're gonna change your relay-based place into a solid state, right? They're learning these concepts with VEX. So, then what is the next evolution hopefully for our history to take in some of these other questions? Whether it's a robot integrator, what is the student that's in VEX gets done with high school, goes to your campus, learns these further codings, right? Maybe they become the Disney Imagineer, right? That are using a lot of these new entertainment aspects, but also functionality.

So, within that concept then, how do we take what we've learned, progress it into the new, not-created-yet job site? Yeah, I mean, in the short-term, there's gonna be lots of jobs for people who know how to use AI to do their job, right? People today are using LLMs to help them do their marketing job, their sales job, or their customer support job. In a few years, when these humanoid robots are more common, there'll be lots of jobs for robot trainers and robot customizers.

Thank you for your attention and engagement. We look forward to seeing how these discussions evolve and shape the future.

The worry is that, a little bit further down the road, if we do reach artificial super intelligence, the LLMs will be better than us at everything. And at that point, we better hope that they find us cute because they're not gonna find us useful.

Hi, sir, thank you for presenting today. It was really cool seeing the different experiments that you're doing. So, I am, yeah, over here. Yeah. Sorry. Had a bit of a technical question for you. So, I'm teaching both engineering and math at the high school level. So, I like my students to just kind of, like, I want them to understand how these things work, you know, what are the mechanisms underlying all these things.

So, with LLMs, I just wanted to ask, like, my understanding, they're basically, the underlying algorithm, it's basically performing high-level, multidimensional matrix math, correct? Like, I'm imagining something like millions and millions of sifters operating per second, and that's how it filters down its responses to you. And I was trying to bridge that over to my algebra class and show them a very rudimentary matrix example. It was really cool to see the example sentence that you gave at the beginning of this presentation because that's essentially what I showed my algebra students. And so I was just curious to ask you more about that.

Yeah, I gave a talk here two years ago where I had a little bit more about how LLMs work. Yeah, they're deep neural networks, and so a neural network is a lot of matrix spectrum multiplies interspersed with non-linearities. The transformer architecture is what led to the breakout of LLMs as being these wildly successful things. But going into detail about how the computation works, that's another talk. So, I'm gonna have to stop here. Was there another question?

That's all we have time for Dr. Touretzky, but thank you very much, please.

(audience applauding)

(upbeat music)

VEX Robotics Educators Conference.