A Voice from the Back of the Room
Mark Guzdial recently posted another thought-provoking piece, this one about how teachers are biased toward assessing a class’s progress by their interactions with its brightest students, and how important it is to assess understanding systematically to find out what’s actually going on. We’re still wrestling with this in Software Carpentry: what can we put on the web to help students determine whether they have understood a particular topic and are ready to move on?
- Multiple choice quizzes can only be done once: if you don’t get the right answers the first time, go back to review the material, then try the quiz again, all it’s really testing is your ability to recall that the answer to #3 is ‘c’.
- Small programming problems don’t work either. Even in simple cases, there are always many right (or at least workable) answers. Novices are, almost by definition, not able to see whether their solution can be mapped onto the instructor’s; all they see are the differences, and they come away thinking, “Oh, I didn’t get that,” rather than, “Huh, I guess there is more than one way to do it.” This can be ameliorated a bit by providing sample input and output, so that they can test their programs, but that doesn’t help them figure out why theirs is right or wrong.
What else could we do? How else could we help you figure out whether you’re ready to move on to the next topic, or ought to review this one again?
I don’t have any creative novel solutions to these problems (except for the one vague attempt at the end), but I do think that some form of the two you mentioned can accomplish at least the core goals.
Multiple choice:
Presumably it would pretty easy to do two things to avoid the basic problem you mention. First, randomize the order of the answers to avoid the most trivial kind of retention from the first round. Second, and I think more importantly, write more questions than will be presented and then randomly select questions for each round either with or without replacement (the latter being predicated on a login or a cookie of some kind). This might seem a bit tedious, but this would be an area where you could easily involve secondary contributors who right questions that are then submitted through a workflow of some kind for approval. You could even encourage the writing of new questions as a exercise for students.
Small programming problems:
I actually find these quite useful. I got a lot out of Project Euler when I was playing around with it a few years ago and at its most basic all they provide is a check of whether your end result is correct. This can of course be frustrating if you can’t figure it out and this may be more of an issue for you “back of the room” students.
Leveraging the social aspects of the web:
One thing that Project Euler did to improve on this simple, did I get the right answer approach, is the existence of a forum for the discussion of answers to each problem. On this forum people post their solutions and discuss them with each other. This really helps with the “there is more than one way to do it” and also helps people see better ways to accomplish the same task and why some may actually be better than others.
Fill in the blank?:
What about having a chunk of code with some pieces missing? This might let you constrain the students in such as way as to make it difficult to get a meaningfully different solution to the problem, yet still provide a solid test of their knowledge in the area of interest.
I think Ethan’s right, particularly about the multiple choice. That’s how the `Concept Tests’ in physics instruction is done these days; several multiple choice questions to test to see if the students have gotten the basic point in the lecture-ette yet before proceeding, and you necessarily have to have a couple. It’s more work, of course, and frankly I find coming up with good questions the hardest part of the whole endeavor.
As to the code tests — if you find a general solution to the problem of `look at a code snippet, see if it is Doing The Right Thing, and if not, offer useful suggestions’, then you’ll have a very significant contribution to the field indeed. If you do limit yourself to a line or two of code as Ethan suggests, then the problem becomes much easier. SQLzoo.net, for instance, does this in teaching SQL and it works quite well because of the restricted domain. This does not, of course, test their ability to write a program to solve problem X, but it might test whether or not they understand the very particular coding issue at hand.
I very much second Jonathan’s point. I teach economics and without frequent feedback from students with clickers when asking conceptest-style questions I’d have MUCH less of an idea of what my students know.
On conceptests, http://www.youtube.com/watch?v=lBYrKPoVFwg is nice short intro. For more info, see http://www.youtube.com/watch?v=WwslBPj8GgI and http://www.compadre.org/Repository/document/ServeFile.cfm?DocID=241&ID=4990 .
I also use Team-Based Learning; see http://teambasedlearning.apsc.ubc.ca/ for details. For it in action, http://magenta.cit.utexas.edu/largeclasses/#tbl .
I wished I had words of wisdom on how CS might be taught with these methods.