(bright music)

Please welcome Dr. David Weintrop from the University of Maryland. He has a PhD in learning sciences and a background in software development. David has done a lot of work with VEX Robotics, so if you are interested in seeing any of his previous work, you can find that in your PD+ video library, or you can check out some of his research on research.vex.com. Please give David a warm welcome.

(audience applauding)

All right. Hello, everyone. Good afternoon. Thanks for having me. I teach a lot of undergraduate classes, so I'm actually pretty comfortable with people in the back not paying attention, but I will be cold calling on people. So if you're not paying attention, that's what you're signing on for.

Okay, so I want to spend some time today shifting gears a little bit from the talk that we just heard, but I want to share a little bit of the research that I've done, particularly around block-based programming. To start, I just want to share a big picture of who I am and the types of things that I think about, the types of work that I do. And then from there, dive into some of the research that I want to share.

Broadly, my research is focused on creating effective, engaging, and equitable computational learning experiences for all learners. What that looks like is I've done a bunch of work I'm going to share today about block-based programming, trying to understand the strengths and weaknesses, affordances, and drawbacks of introducing students to programming in block-based environments. I've done some work designing elementary school robotics curricula, looking at ways to integrate computational thinking into math classrooms.

A big project has been almost a decade, collaborating with some large school districts, designing a middle school computer science curriculum called Scratch Encore. The idea behind Scratch Encore is situating meaningful, substantial computer science conceptual knowledge in middle school in ways that resonate with learners and draw on their cultural knowledge and prior resources.

I also have a high school data science curriculum. This is some new work that we're doing, looking at ways to situate data science in high school students' lived experiences. High school students create and consume data all day every day, so how can we use that as a way to bootstrap foundational data science ideas?

Finally, a project called Impact Libraries, Improving Assessment of Computational Thinking in Libraries. I mostly highlight that to say, while I'm primarily focused in K-12 spaces, I also hang out in libraries thinking about libraries as community resources to introduce big ideas of STEM to learners.

As I go through, I'm going to spend today's talk talking about block-based programming. Partially, I wanted to mention these things in case any of those things pique interest, like, "Oh, tell me more about data science," or, "Oh, I was talking to a teacher and we're looking for some middle school curricula."

Another thing I wanted to say upfront is I do a lot of collaborating with teachers, and I always talk to teachers about, and encourage them to, when they can take a breath and step back from their day-to-day, try putting a researcher hat on and think about their own practice in terms of what they are learning from students, reflecting on their own practices of what's effective, what's not. What I'm going to present today is a very formalized academic version of these things, but these types of little experiments teachers are doing all day every day.

So, one, encouraging you as a teacher to think from this researcher perspective. And then two, please share those things that you've learned with peers and also back with researchers.

Because a feature of my job is that I have the bandwidth to kind of go deep into some topics that you, as teachers, will find challenging, given that things are moving at a million miles an hour. I love hearing from teachers about things they've noticed and ideas they've had, such as the API Can Code project. A lot of the ideas for this high school data science curriculum came from talking with teachers about questions they had, things they were trying, and their lived experiences and what they've learned from that.

So, if at times my talk starts to get boring, you can let your mind wander to the types of research questions you'd be interested in asking about your own classes. But don't let it wander too far because I'll call on you, and then where are we?

Okay, so I'm gonna talk today about block-based programming. I'm gonna assume folks are mostly familiar with block-based programming, but if you're not, this is the Scratch programming environment.

Music Cue

So a few things to notice here is a short video writing a program. First, we're gonna drag a block into our coding area. It says move 10 steps. I click on it, and our little Sprite moves 10 steps. I'm gonna hop over to looks and pull out a block that says change color effect by 25, snap it into place. And now when I click, my little cat moves and changes colors. This is very cute and exciting. Move over to control. I'm gonna take a repeat block. It says repeat 10. The shape of the block changes to wrap around those two commands. And now when I click on it, those two commands run 10 times. So, we have iteration. And then finally, drag out a block that says When space key pressed. And now if I hit the space bar, my cat does this little program.

Music Cue

So a few things to notice about what we just saw here. First, we have user interaction, right? There's a keyboard input, we have iteration, we have multiple steps running in order. We did all of this in about 10 seconds and we didn't touch the keyboard, right? And so thinking about what it looks like to get kids up and running programming doing cool, interactive, engaging things. Block-based programming has a whole bunch of affordances.

Another reason I'm talking about block-based programming is this is VEX VR. It is now possible to program VEX robots using VEX VR. There are also some neat ways to integrate VEX VR into other Python programming things I'll talk about a little bit. I know that there are a couple of sessions on VEX VR today. There was one this morning that had 20 people sign up and about 45 people show up. So there's lots of excitement in the space, and for those of you who haven't played around in VEX VR, I encourage you to do so.

But the big question that I want to talk about today is this idea of, to block or not to block? Is block-based programming a good idea? When I was initially starting to explore this, it gets back to what I was saying in terms of teachers as the best sounding boards for asking important research questions. When I was starting this work, I would meet with high school computer science teachers and ask them, like, oh, are you using block-based programming? What do you think of block versus text programming? Is it working, is it effective? And the response I always got when introducing the study I was doing was, this is a question I want the answer to. Like sometimes I do block-based programming and I'm not sure students are into it.

Especially since tools like Scratch, these block-based environments, are very colorful, playful, and designed for middle school students to be fun and engaging. Putting these tools in front of high schoolers or even undergrads presents a fundamental tension where there's an assumption because it's colorful and cartoony that it's a toy language, a trivial environment.

Thank you for your attention and for being such an engaged audience. I look forward to hearing your thoughts and questions.

And so the question was like, is this effective? This was something that teachers were really invested in. I spent three years working in a bunch of different classrooms in a large urban school district to try and make sense of whether or not block-based programming was a good idea.

The first thing that I did was interview a bunch of students. These were high school students in an intro programming class. I more or less just asked them, "What do you think of block-based programming? Do you like it, did you not like it, is it easier or harder?" Overwhelmingly, students thought that block-based programming was easier than text-based programming. They said things like learning Java is more complicated than blocks. In Java, there are no blocks to help out; it's basically done from scratch. I assume, no pun intended. This isn't surprising, but it shows that high school students acknowledge and recognize that these drag-and-drop environments are easier to use than conventional text-based environments.

Digging a little bit deeper, as a former software developer with a lot of experience in the design of programming environments, I tried to understand what makes block-based programming work. Students talked about how blocks are easier to read. In theory, a valid block could be used in a program, making programming more narrative, like reading stories rather than parsing syntax. They mentioned the shape and visual layout, noting how oval pieces fit inside oval slots, and how notches and things snap together in satisfying ways.

Students also found block-based programming easier to compose. For seasoned programmers, we often underestimate the barrier of trying to find curly braces, semicolons, and square brackets. These can be intimidating and unclear on the keyboard, but dragging and dropping goes a long way. The idea of browsability is a powerful one. One student said the blocks jog your memory. In psychology, we call this recognition versus recall. Instead of knowing ahead of time what's possible, you can browse through available blocks and think, "Oh, that one sounds pretty good," then drag it out and away you go. This is opposed to staring at a blank sheet and needing to know everything available within a programming language or library.

Students also talked about support for trial and error. If any of you have done coding before, you know how critical trial and error is as a way to tinker with things. Students perceived block-based programming as a more tinkerable, playful environment. In contrast, in Java, you need to know what you're doing; there's no trial and error. But in blocks, you can, which is fascinating.

Prefabricated commands allow one block to do a whole bunch of things, whereas the grain size of programming languages is much finer. Lastly, a characteristic of Scratch and similar environments is that the program you're writing is live. You can see the outcomes as you go, which again ties back to the idea of tinkerability. What's interesting about some of these things is the visual outcomes, support for trial and error, and versions of browsability.

Thank you for your attention and interest in this topic. I hope this exploration of block-based programming has been insightful and informative.

These are not unique to block-based environments, but these are characteristics that students identified as making them easier to learn. It is not all sunny skies. There's also some perceived drawbacks. They saw block-based programming languages as potentially less powerful, right? Blocks are limiting. There's not a block for everything. Talk about slower authoring, dragging and dropping at some point. Typing is a lot faster. And then students would also talk about things like inauthenticity.

I think, again, these are student perceptions, so I don't mean to dismiss them because these are really important things to address. But at the same time, we have block-based Python, we have block-based C, right? The idea that block-based is inherently less powerful, inherently limiting is a student perception, but it need not actually be the case. And in many cases, isn't the case.

All right, so this is a first attempt at understanding student perceptions of block-based programming. We can see that students perceive it as easier. I attribute that ease to particular characteristics, but they also saw some drawbacks.

Okay, so that's part of an answer to the secret sauce of block-based programming. But the big open question is how does it compare to more conventional text-based languages?

I spent two years working in a series of high school classrooms and designed a classroom-based three-condition study where I had a block-based version of a programming environment, a text-based version of a programming environment, and then a hybrid environment. The design of the study was for this, for what I'm gonna present, I was working in one school with one teacher all in the same classroom and essentially worked out where these students were fourth period, these students were seventh period, these students were eighth period.

It started on the first day of school. So starting on the first day of school, we ran a five-week curriculum where students either learned in a block-based environment, hybrid environment, or text-based environment.

To give you a sense of what these environments looked like, it was a customized version of the pencil code environment. This is the block-based environment. This is a conventional typical block-based environment. You can browse through the blocks palette, drag out a block, drop it into place, modify it, and run it. We get our little turtle here to draw this nice little spiral diagram. This is what it looked like for the fourth period students to learn, went through a curriculum using this environment.

This is the text-based environment. Whereas before we had blocks, like the blocks palette and drag and drop, now it's just a conventional text editor. But when you run, you get the same output. Importantly, and it'll be clear on the next slide, character by character identical. This is what the seventh period class students worked in this environment.

In eighth period, we came up with a hybrid environment where we have a text editor, but we still have the browsability and some of the drag and drop features from block-based. Now you can drag a block in. But when you drop it, it gets dropped in as text and you can modify it. Again, if you run the program, you get the exact same output. Ta-da, spiral.

Again, fourth period, seventh period, eighth period, same teacher, same school, same class, same curriculum, same time on task, five weeks, starting the first day of school. The question was, in terms of measuring student understanding of programming after five weeks, which students are gonna perform the best? Which students will have learned the most computer science?

One other important thing to talk about just for a minute is assessment. If we're gonna make claims about student learning, we need a way to measure student learning.

We designed an assessment we call the commutative assessment. Here's an example question from the assessment: What will the value of x and y be after the script is run? Set x to 10, set y to 10, or set y to x, x to x + 5. Think about it in your head. This is one example of a commutative assessment question.

Here's another example of a commutative assessment question. The idea here is that every question we asked had both a code snippet and a multiple-choice answer. The code snippet could either be presented in a block-based way or a text-based way. We would alternate versions of the code, so students would answer two block-based conditional logic questions, two text-based conditional logic questions, two block-based logic questions, and two text-based logic questions.

In designing the multiple-choice responses, we dove into the literature to see what misconceptions had been identified. For example, there's some work that was done 50 years ago, finding examples of students who would think that they wouldn't evaluate the values of variables but instead think of them as strings. Or this last one, E, which I think is super interesting, is the idea that students might think that a variable kind of has memory and can hold multiple values at once. It turns out to not be the case, but this is a reasonable potential interpretation.

The idea was that we could not only see how students are performing in terms of correct or incorrect answers, but we could start to unpack the types of misconceptions that students had and whether or not that correlated with the different types of programming environments they were using.

We did a pre-assessment on the third day of school. We gave them this pre-assessment to see how much they knew about computer science. Then, after five weeks of working in blocks, text, or a hybrid, we ran the same assessment again. The first thing to mention is that on the first day, or the third day of school, these dots represent the three different classes. As researchers, this is what we want to see. It essentially says there's no statistically significant difference between the three classes. This means that in terms of prior knowledge, all three classes are starting at the same place.

Okay, audience participation time. If you think students in the blocks condition performed the best out of these three classes, hold up a one. If you think students in the hybrid condition performed the best, hold up a two. If you think text did the best, hold up a three. Maybe hold up a fist if you think there's no difference. So, what do we think? Blocks, hybrid, text. Some hybrids. I don't think I see anyone holding up a three.

Survey says, ta-da. The first thing to note is all these lines are positive; all students learned. This is good. More importantly, apologies for the statistics, but what this is saying is there is a statistical significance between these performance measures, and that difference is coming from students' performance on blocks versus students' performance on text. This means students performed statistically significantly higher in the block-based condition than in the text-based condition. Students in period four scored higher on the assessment after five weeks than students in period seven.

We can also unpack this a little bit and look at how performance differed based on the types of questions they were asking. We see students in the blocks condition, blocks is blue, text is red. Students in the blocks condition scored significantly higher on conditional logic, functions, and iterative logic.

We can unpack that a little bit and start thinking about what are the characteristics of conditional logic functions and iterative logic that we don't see with something like variables. This is where things like curly braces start to show up in challenges around parsing what programs look like. Essentially, what this shows is after five weeks, students in the block-based condition performed statistically significantly better than students in the text-based condition.

Thinking back, we've learned something, and this is good; this is progress. This is one data point showing that maybe block-based programming does have a role in computer science classrooms. This is with high school students who were in a year-long Java class, and the first five weeks were spent in block-based programming.

I want to take a quick side step into another study that we were able to run to further explore the limits of this. I'm guessing most people here are familiar with the AP Computer Science Principles course. This was introduced not that long ago. The idea here is this is an advanced introductory computer science course, which is a little bit of an oxymoron, but it is what it is.

One of the fascinating things about the AP Computer Science Principles exam is that instead of prescribing a specific curriculum or programming language, different organizations were able to design their own curriculum. This meant different curricula used different programming languages. Students in an AP Computer Science Principles class in some schools would learn in environments that looked like this, while others would learn in environments like that. Some environments allowed students to choose or move back and forth between them.

This poses an interesting problem because, for those familiar with AP exams, there's also a pen and paper exam that students have to take at the end of the year. The question becomes, given that we don't know what students are learning in, and we just showed you that this actually matters, this is consequential, how would you assess this?

Well, this is what the AP board came up with, which I think is fascinating. They invented their own programming language, of course. This is what it looks like, taken from the test guide, showing what you can do in the programming language. They wisely acknowledge that this is a text-based environment, so they also needed to invent a block-based environment. On the test, the pen and paper test that students were taking, they'll get asked questions that sometimes would look like this, interpret this program, and sometimes look like that, interpret that program.

Again, looking character by character, they're identical, save for parentheses and curly braces being replaced with ovals and circles. There's some consistency there, though I'm not sure why the sum is a rectangle and this is an oval, but so it goes.

Okay, so I just showed a slide that said we did the research, and it said that block-based is better than text-based. Now we have a new question: is this thing better than this thing? I'm going to ask you guys to vote again, but before I do, let's come back to this slide. It says these are all the things that students said were better about or what made block-based programming more accessible and easy to learn. If we look at those characteristics and talk about them in terms of this AP computer science exam, only this one is left, right? Character by character, it's the same. They're not writing any programs, they're not browsing things.

Thank you for your attention and participation. Your insights and questions are invaluable as we continue to explore the role of block-based programming in education.

So it's only the shapes and visual layout that's the difference. Okay, let's vote. If you guys think that students perform better on, oh wait, let me talk a little bit about methods. We collaborated with Code.org and had around 5,000 students across the country take a short survey. We created a version of the cumulative assessment that asks some questions in this environment and some questions in that environment.

Okay, so if you think students perform better in the block-based version, put up a one. If you think they did better in a text-based version, put up a two. And if you think there's no difference, and this is a silly question, why are we even doing this, just hold up a zero.

Getting a lot of ones, which is guys a lot of ones, but how could that be? It's like what, there's nothing here. It's just the...

Well, you guys are correct. Students in the solid color, these are students in the block-based condition, and we can say overall students scored statistically significantly higher in blocks versus text. Across these different demographics, students scored statistically higher, which to me is wild, right? As a researcher, I was like, no, no, no, I did the research, and kids said this, and it's like, I did not expect there to be a difference.

It's not just that there was a difference, and this is really, really important. The idea that, or the finding that students from populations historically excluded in computing contexts, and by that we mean women and underrepresented minorities, which for this study, Code.org defined as Black, Hispanic, Native American. Students in those categories not only scored statistically significantly higher, but the gains they saw from the block-based to text-based were larger than their peers.

Which is one, a little wild, but two, talks about how consequential these decisions are, right? It's not just an exercise in what do I think is easiest to teach with or what are the preferences of kids, but this has meaningful substantial impact on how students perform. We know that performance in classes correlates with and leads to students having a high level of confidence, a higher level of interest, and persisting in computing, whereas they might not have done so otherwise.

Okay, so we looked at the initial study comparing blocks versus text and found that blocks were better than text. A second part of the study was this idea of, so we showed that over five weeks students scored higher in block-based than text-based. But in this case, and in many cases, educators think about block-based environments as introductory tools and on-ramps with the hope that as they move forward in block-based environments and gain confidence, skills, and knowledge, the plan is to have them transition to a text-based environment because it has a higher ceiling.

If they want to get into the workforce, this is what it looks like. If they want to do the most, build the most gnarly autonomous robot, it's gonna be an environment that looks like this as opposed to an environment that looks like that. So the question becomes, so I showed you the study design initially. Right on the first day of school, we spent three or five weeks in blocks, text, and hybrid. After five weeks, all of the students transitioned into Java and spent the rest of the school year programming in Java. The first 10 weeks we followed them in the Java classroom to try and understand whether or not students who came out of the blocks-based condition scoring higher on our assessment meant that they were ahead and kind of accelerated their learning in Java.

All right, I think this is the last time I'm gonna have you guys vote. So this is where I left you guys off. We see our statistically significant divergence between blocks and text.

Thank you for your attention and participation. Your insights and engagement are invaluable to our research.

Hold up a one if you think after 10 weeks in Java text, hold up a one if you think blocks scored the highest. A two if you think hybrid kind of pulled ahead. Or a three if you think that the text environment pulled ahead. Or a fist, if you think there's actually a difference.

Ta-da. What this says is there's actually no statistical significance between these three. All three classes essentially didn't score or aren't meaningfully different from each other. So what we saw is at the beginning of the school year, they started in the same place. They diverged as they went after five weeks and then converged again after 10 weeks in Java. I tricked you guys, right? Everyone's like, oh yeah, I know how the story's gonna end.

So a couple of things to say about this. One version, one interpretation, which is not the interpretation that I adopt, is, well if you begin and end in the same place, what's the point of doing this blocks-based thing in the middle? I'm gonna show the next couple of slides that argue for why that's important.

The second thing to mention that I think is really critical, and this is something that I think folks or educators might intuitively think is the case, but the more you think about it and interrogate it, it starts to fall apart, is this idea that transfer comes for free. In education, we talk a lot about transfer, learning something in one context and applying it to another. Education researchers know that transfer is extremely difficult to do, to find, to prove, to show.

What this is showing is that even though students showed a higher level of performance and a higher level of conceptual understanding of programming ideas on these assessments, when they moved to the Java programming environment, those ideas didn't carry over. In interviewing students later on, they would see where, ah, like we understand conditional logic here, we do really well. And then when it comes time to learn conditional logic in Java, it's like starting from the beginning again. After they've learned it in Java, they go, "Oh wait a minute. It's just like it was before in block-based," so that it was only after they relearned it could they make the connections.

Part of what we're seeing here is this lack of divergence is empirically showing that this idea of transfer doesn't come for free. Another really important thing, and this is a spoiler, we're gonna end the talk. For the sake of the study, the teacher didn't do anything. There were no pedagogical moves to support learners in transitioning from this environment to that environment. As a researcher, I intentionally was very hands-off, like I let the teacher teach how she wanted to teach. By the time she got to teaching in Java, and again, she taught all three of these classes. She taught the blocks condition, the hybrid condition, and the text condition. She taught them all the same way, which means she didn't put up slides that said, oh, here's conditional logic from blocks from the first five weeks. This is what it looks like for Java to kind of do this bridging from the context where they had already learned these ideas into the new context.

She didn't do that because she was an extremely generous and extremely talented, committed educator, but wasn't gonna teach three versions of the same thing. Wasn't gonna make three sets of slides to support that. To me, as an education researcher, it says there are open questions as to what it looks like to support learners as they move from one environment to another.

Look at the time, knowing VEX has some ideas in that space. I just wanna quickly show a few other things. We also asked students, we gave attitudinal surveys.

We asked students questions related to confidence, sense of belonging, interest in future computer science programming, and similar topics. Let me show you what this looked like. For example, we had an aggregate confidence score. We asked students on a five-point Likert scale three questions related to confidence and merged them together to come up with an aggregate confidence score.

What we found was interesting. From the beginning of the semester to the midpoint, students in the block-based condition were significantly more confident after learning block-based programming for five weeks. However, they became significantly less confident after spending 10 weeks learning in Java. This suggests that when things were going well and they were performing well on tests, their confidence was high. But the transition to Java became more difficult, and their sense of how well they were doing started to decline.

We can see the same pattern emerge across an aggregate enjoyment score. Again, three questions were asked, including whether programming is fun. It seemed that programming became less fun in text, but interestingly, it became more fun for students who had been doing text-based programming for longer. The difficulty of programming was also noted.

Finally, regarding future computer science interest, we observed opposite trends. Students in blocks thought they were more interested, but the transition to text became more challenging. However, once they moved to Java, they became more interested.

A couple of things to highlight here include the unsatisfying takeaway that doing this type of research is messy. It's important to consider when these transitions take place over the course of the semester, especially relative to things like add/drop deadlines. Some interesting research from Carnegie Mellon showed that students who excelled in CS1 would systematically leave CS2. This is particularly relevant when looking at students from historically excluded populations in computing and the decisions they made.

For me, these are open questions and areas to pursue to more deeply understand the intersection of representation and how it affects students' feelings about participating in computing, both in the short term and long term.

To wrap up, we saw that blocks were better than text, but the transition still presents open questions and work to be done. In the study I shared, I showed an initial version of blending block-based and text-based programming by letting students drag and drop blocks into a text editor. This is not the only solution, and it is not the solution that VEX is pursuing.

For those who haven't seen, VEX has a new feature called VEX Switch mode. This is a fully featured block-based environment with switch blocks that allow you to type in code inside of it. You can be coding in blocks but write a single line of Python. You can also right-click on a block and convert it from its block-based environment to a Switch mode block.

Thank you for your attention and interest in this research. We look forward to exploring these questions further and finding effective solutions for teaching programming.

So as you can author a program all by dragging and dropping and then right-click and turn it into these hybrid blocks to modify them from there. The idea is to explore the transition of what it looks like. The same way the teacher did not do that as part of the study. There were no explicit scaffolds from blocks to text. VEX is exploring solutions to that problem.

If this piques your interest, Dr. Jimmy Lin will be giving a talk on Switch mode, transitioning from blocks to text on Wednesday. This was Jimmy's dissertation work. He spent the last two years looking at exactly this problem and working closely with VEX around designing solutions to understand how to support students as they move from one to the other.

Yeah, so I'd like to end by making a pitch for you guys to come back and listen to Dr. Lin share his research, which is, I think, a nice extension and complement to what I shared today.

Thank you for listening, and I look forward to your questions.

(audience applauding)

Any questions from the audience? Speechless. I get it, I get it. I was gonna say that. Mind-blowing. I was gonna say that. Do you have any guesses as to what... Hi. (audience laughing)

Yeah. What would you guess would've happened? I know that this is not the work of research, but if appropriate scaffolds had been put in place to support that transition, do you anticipate things might have ended up differently?

Yeah, great question. And we've done some other work that I didn't present today where VEX Switch mode supports some of this. We've done studies in classes where students themselves can decide if they want to use blocks or if they want to use text and move back and forth between the two types of representations on their own. This kind of experiment has been done by a couple of people in different environments and grades and contexts, including middle, high school, and undergrad.

One of the robust findings from that, which I find interesting, is that they find over the course of the semester, students on their own consistently at the beginning of the year will spend time in the block-based environment, but by the end of the semester have decided on their own to start spending more time in the text-based environment. So they see this transition of, they themselves, like as their confidence with block-based programming increases, they transition to text-based, but in an environment where they can also move back if they need to. We can see systematic patterns of when they move back, like the first time they introduce a new construct, like, "Oh, I'm gonna use a conditional logic thing for the first time." They'll take their program, move back into blocks, add conditional logic, and then hop back into text. So that type of evidence makes me think that with the right tool design and the right scaffolds, we would see the gains of block-based persistent to text-based environments.

Yeah, thank you, great question.

Hi, I teach both coding and robotics, so I have actually, in middle school, I use Scratch and then in high school transition over to Java. What I find is the Java programs are so much more boring because they don't have that visual output in the same way. It's really hard to find something that supports that. So I was wondering if you have found in any of your research ways that are programs or something that makes some of that Java stuff a little bit more visual output oriented?

Yeah, yeah, just like the liveness of it. What students call visual outcomes. So there are definitely some environments that have things like that baked in. I've done a little bit of work doing that with things like the micro bit where you can kind of program in text and have it immediately show up on a physical device.

In terms of environments beyond that, I know that for the study that I did, we used Turtle Library. Processing would be another example of this, like a super live, reactive, engaging environment. Then, thinking about robotic simulation environments where you can do it virtually as a way to get that same level of satisfaction. Yeah, 'cause seeing robots drive through virtual castles is far more fun and engaging than, like, sorting lists of integers or whatever.

Yeah, great question. And by the way, if other teachers have ideas, please share. We're all in this together, yeah. One of the things that I noticed as you were going through your experiment is that you had the students learning block programming and then jumping into Java, which is a beast unto itself and extremely difficult for beginners. Yep. Would it have been better if it were designed where you went from blocks to Python, which is a little bit more reader-friendly?

Yeah, yeah, yeah. Great question. So part of me, short answer is, I don't know. Interesting question. And thinking about how much characteristics of the language itself contribute to its learnability. I think in some ways. So some of these characteristics I think still hold true. Like I put a lot, I think this characteristic of browsability, especially for novices, the fact that they can kind of see what's available and use that to bootstrap their thinking. I think that's a really powerful component to it.

I also know there's been a bunch of work around syntax errors and how terrifying and generally unhelpful syntax errors are, both for experts and for novices. And so thinking about that being another way to bootstrap programming in text-based environments. Short answer is, I don't know. We should do that. But great suggestion, and I know this is, so we used Java in this study because that was the language that the teacher we were collaborating with used, and it was also a course to prepare students to take the APCS exam. But I also know that increasingly, Python is the language of choice. And so yeah, there might be something there. Particularly decreasing, like syntactic burden of curly braces and other nonsense. Yeah, great question. Sorry for the unsatisfying answer.

Oh, let me put the slide back up. I have the lucky feature of having a relatively unique last name. This is my email address, but even if you just search for me and my last name, you'll find me. Please don't be shy about reaching out. All these curricula are available, the papers that we published are all there. So, please don't be shy.

Yeah, thanks again for coming.

Thank you, Dr. Weintrop.

(audience applauding)

(bright music)