Assessment Redux
The single biggest challenge Software Carpentry faces right now is how to tell what impact it’s having. This is only partly to satisfy funders—as I said back in December, if we don’t know how to tell if we succeeded, we’re going to fail. It would be (relatively) easy to put together a multiple-choice quiz to see how much people have learned about basic shell commands, the syntax of Python, and so on, but that would only address the shallowest aspects of learning. We’re trying to impart some fundamental principles, and what we need is questions that will tell us whether people have internalized them. (As many studies have shown, it’s possible to get a decent score on a quiz without actually understanding the subject matter.)
For example, consider this question about Subversion:
Emmy wants to see what has changed in her working copy since revision 120. The command she should run is:
svn log -r 120svn diff -r 120svn revert -r 120None of the above
It addresses Q05 (“How can I keep track of what I’ve done?”) fairly directly, but not R02 (“Use a version control system”), R03 (“Automate repetitive tasks”), or any of the basic principles. Open-ended answers might do the latter, but it’s hard to come up with ones that don’t lead the witness: asking, “When would you use a version control system?” isn’t going to give us much insight into what people actually think. We could combine a few multiple-choice with a few open-ended, but realistically, if it takes more than 10-15 minutes for people to answer, many (most?) won’t. If anyone can see a way to square this circle, I’d welcome ideas.
You’ve said you can tell fairly well in person if a student “gets it.” Can you reverse-engineer the algorithm you’re using to make that call?
I could imagine doing this via sequential multiple-choice questions, somewhat like you might do in person. Start by presenting a case or problem, like one of your “audience” descriptions. (“Alice and Bob are working on a piece of code together…”) The first question would look for general recognition of the tool needed to solve the problem. (“use a version-control system”) With a correct answer, you’d present follow-up questions to drill down into the details (Which program do you use? How do you start a new project? How do you commit code? How do you fix mistakes or find help?).
If the student was clearly off-track, you could either guide them back on or present a new case. The overall score would depend on the difficulty of the questions the student was able to reach/answer, similar to the computer-generated GRE.
The short answer is “no” — when I watch people to see if they “get it”, I’m usually watching them do very different (domain-specific) things. Part of what I look for is wasted effort: if someone is doing something in five steps that I could do in one, or if they’re backtracking repeatedly, I mentally file them under “doesn’t get it”, then dig in to find out why not.