Blog

We've had a busy nine months: 55 boot camps since we restarted boot camps last September, with 19 more in the next two months. We're taking a break then to catch our breath: we only have one event scheduled between mid-July and the end of August, which will give us time to reorganize our online materials and figure out what we're going to do in 2013-14. The list below outlines things that I'd like to see done during that break. I'm sure you have others; I'd be grateful if you'd add them as comments, along with your thoughts on things we shouldn't do, or should do differently.

Re-think the boot-camps repository and workflow.

This is our biggest task: it must be much simpler for people with middling-weak Git skills to create a space for a new boot camp, find and import teaching materials used in previous boot camps, and offer their materials back to the community.

1. Decide whether to make changes in place, or create a new repo and start afresh.
2. Catalog the material we have now.
3. Re-organize material so that it aligns with the instructors' guide (or rearrange the instructors' guide if that makes more sense).
4. Put generic setup instructions in the boot camps repo, rather than in the main web site (the latter can easily refer to the former), and make it easy for instructors to clone and specialize those instructions. (People have some fairly strong opinions about how this should be formatted; we'll re-open that discussion at the start of July.)
5. Merge the stuff that's in our GitHub wiki pages into the boot camps repository, then stop using wiki pages: our information is already too scattered, and since this is our least-used communication channel, it's the obvious candidate for weeding.

Regularize boot camp operations.

This is the "how" to go with the previous item's "what". Much of it depends on a small SQLite database that cannot be in a public repo (since it includes personal identifying information).

1. Ensure that everyone is aware of the code of conduct, no matter how they sign up.
2. Make sure people know that anyone can use our content (which is all CC-BY licensed), but the Software Carpentry name and logo are trademarked, and that there are some restrictions on their use (discussed in these two blog posts).
3. Create a unified online checklist for each boot camp with to-do items for organizers and instructors. (There are currently checklists in two different repositories, which are inconsistent with each other.)
4. Create a mailing list per boot camp, plus a host email address that forwards to the organizer(s).
5. Appoint an experienced lead instructor for each boot camp so it's clear who's responsible for sorting out who teaches what, etc.
6. Incorporate a pre-assessment questionnaire into registration (or at least for those who get a seat).
7. Make immediate and delayed post-assessment a one-click process.
8. Make in-camp assessment more systematic (i.e., standardize a set of questions that should be asked at every boot camp, and a way of reporting the answers).
9. Make it easier for potential instructors to find boot camps, and for lead instructors to find partners.
10. Archive waiting lists from boot camps so that people who didn't get in last time are first in line the next time. We could potentially keep track of no-shows as well: someone who has registered, but then fails to attend without explanation, should probably be put at the back of the line for future boot camps, but this could be problematic to implement.
11. Automate synchronization of boot camp announcement pages on the web site, EventBrite registration pages, boot camp GitHub repositories, mailing lists, and GitHub tickets.
12. Collect feedback on instructors to help them improve their teaching (and then actually give it back to them). This will have to be done carefully—we have no wish to encourage a "big brother" atmosphere—but it will help maintain and improve quality.
13. Finish, deploy, and publicize the Windows installation script and the "check my machine" scripts.

Line up our next eight months of boot camps.

We need to book things further ahead, so that we're scrambling at the last minute to find instructors less often. We also need to start targeting conferences and other disciplinary clusterings, rather than doing things geographically.

1. Contact all previous boot camp hosts and ask if they'd like a return visit.
2. Since demand now outstrips supply, we should probably also think about some sort of bidding process, e.g., sites where people are doing instructor training or are willing to help out with boot camps elsewhere get priority.

Finish the instructors' guide.

We desperately need more instructors; we also need to do more to ensure that people are teaching the same (kinds of) things, or at least are aware of what other people are teaching.

1. Create a concept map for each section. (None exist yet.)
2. Create challenge problems for each section. (About half are done.)
3. Draw 250 diagrams. (I'd rather stick a spoon up my nose, but it needs to be done.)
4. Create a printed version (because some institutions still take books more seriously than web sites).

Simplify contribution to the web site.

If we want more people to contribute more content, it needs to be a lot easier to update a page or write a blog post. In the long term, this probably means either building something that integrates with the services we use (like EventBrite and Google Calendar), or switching to a hosted customer relationship management system, but that's not going to happen this summer.

1. Start using Pelican (or maybe Nicola) instead of our home-grown tool.
2. Stop using sub-modules for the 3.0 and 4.0 material (which probably won't ever change again).

Rationalize the mailing lists.

Dreamhost (our ISP) supports Mailman, but doesn't give us access to Mailman's command-line scripting interface (because mail is hosted on different machines than shell accounts). We therefore have to manage list membership manually, which produces all sorts of minor inconsistencies. We're planning on switching ISPs anyway (Dreamhost is increasingly unreliable), so we have an opportunity to fix some of these problems.

1. Make sure the ISP we switch to offers scriptable mailing lists.
2. Integrate mailing list management with our roster.

Regularize instructor training and badging.

We're now running the instructor training for the fifth time, and we've learned a lot about what works and what doesn't.

1. Put a rough curriculum on the teaching web site.
2. Switch from a WordPress blog to a compiled web site using the same tooling as the main web site.
3. Clarify the certification process: people must complete the training and teach at least once under the supervision of an experienced lead instructor.
4. Start generating PDF certificates for instructors as well as digital badges.

Certify learners.

We've been asked several times to provide certificates of completion. We would rather provide certification of accomplishment, i.e., give people something only once they've demonstrated some level of proficiency, rather than for sitting through two days, but we can certainly do both.

1. Choose and publicize proficiency assessment criteria.
2. Automate production of PDF certificates and digital badges for both completion and proficiency.
3. Make sure that whatever we do for proficiency integrates with the driver's license being developed in conjunction with the Software Sustainability Institute.

Miscellaneous

• Create better publicity materials. We retired our 90-second pitch months ago because it no longer represented what we actually do. We need to create new ones aimed at potential learners, hosts, instructors, and backers. We also need to write a paper that people can refer to when they want to cite us—our Best Practices in Scientific Computing paper isn't quite that. (Software Carpentry: Lessons Learned is a start toward this.)
• Break the assets repository into several pieces. At the very least, assessment materials, papers, and checklists should be in their own repos.

In January 2012, John Cook posted this to his widely-read blog:

In a review of linear programming solvers from 1987 to 2002, Bob Bixby says that solvers benefited as much from algorithm improvements as from Moore's law: "Three orders of magnitude in machine speed and three orders of magnitude in algorithmic speed add up to six orders of magnitude in solving power. A model that might have taken a year to solve 10 years ago can now solve in less than 30 seconds."

A million-fold speedup is pretty impressive, but faster hardware and better algorithms are only two sides to the triangle. The third is development time, and while it has improved since 1987, the speedup is measured in small percentages, not orders of magnitude. For most scientists, getting the code to do the right thing is now a bigger bottleneck than its running time.

That's where Software Carpentry comes in. We teach scientists (usually grad students, since they have both the need and the time) what they ought to know before they start working on large programs, creating web services, or any other leading-edge work. Our hope is that if we give people basic skills, they'll be better able to take advantage of more sophisticated things.

Between January 2012 and July 2013, over 100 volunteers will have run 92 two-day workshops for over 3000 scientists. A typical two-day curriculum is:

• Day 1 morning: Introduction to Unix shell. We show participants a dozen basic commands, but the real aim is to introduce them to pipes, loops and history (to automate repetitive tasks), and the idea of scripting.
• Day 1 afternoon: Version control, with an emphasis on collaboration and reproducibility. If time permits, we also give them a quick intro to regular expressions or something else that's relatively easy.
• Day 2 morning: An introduction to Python, the real goal of which is to show them when and why to develop code as a set of comprehensible, reusable functions. Some boot camps use R instead, and we expect that to become more common.
• Day 2 afternoon: Testing, focusing on the idea of tests as a way to specify what a program is supposed to do, and how to to structure unit tests using an xUnit-style library. This is probably the weakest part of our standard two days: we've got the "how" down cold, but the "what" for scientific software is still very much up in the air.
• Day 2 afternoon: Number crunching using NumPy, basic SQL, or some other "beyond mere programming" topic that suits the audience.

As the comments above suggest, our real aim isn't to teach Python, Git, or any other specific tool: it's to teach computational competence. What we've found, though, is that we can't do this in the abstract: people won't show up, and if they do, they won't understand. We try hard to start with the particular, and to show them that yes, this stuff actually is useful, and then bring in more general stuff.

We nominally aim for 40 people per workshop, and are always grateful for local helpers to wander the room and answer questions during practicals. We find workshops go a lot better if people come in groups, e.g., 4-5 people from one lab, half a dozen from another department or institute, etc., so that they are less inhibited about asking questions, and can support each other afterward. (It also produces much higher turnout from groups that are usually under-represented in computing, such as women and minority students.) We use live coding rather than slides: it's more convincing, there's more lateral knowledge transfer (i.e., people learn more than we realized we were teaching them by watching us work), and it makes instruction a lot more responsive.

Our instructors are all volunteers, so the only cost to host sites is travel and accommodation. All but a handful of our instructors are working scientists themselves; that, plus live coding instead of slides, ensures that attendees get lots of "how" as well as "what".

We also run an online training course for would-be instructors. It takes 2-4 hours/week of their time for 12 weeks, and introduces them to the basics of educational psychology, instructional design, and how these things apply to teaching programming. It's necessarily very shallow, but it's still more than most university faculty ever get...

Results have been very good: we had two independent evaluations done last spring (one by Prof. Julie Libarkin at Michigan State University, the other by Dr. Jorge Aranda at the University of Victoria), and both found that people actually are learning useful things. What we're struggling with now is showing that this translates into them doing more and/or better science (the holy trinity of "novelty, efficiency, and trust").

Here's what else isn't working (or isn't working well):

• Our "host site pays costs" model makes us nearly self-sustaining financially, but only if you exclude the costs of 1.5 people in the middle of things doing coordination. Since I'm the 1.0 of that 1.5, I'd like to find a way to regularize that size of things...
• Learners come to us with a very wide range of prior knowledge and abilities, and as a result, we figure that 20% are bored and 20% are lost at any time. We'd like to start offering separate boot camps for complete novices and can-already-kind-of-program people, but we can't rely on them to self-assess their ability (they don't know what they don't know), and if we give them any kind of proficiency test, it scares away the ones with weaker skills who need us most.
• We need to start teaching data management and how to use the web to do science better; the hard part is to figure out what. We can talk generalities about data management, but they want tools and procedures. As for web programming, all we can show them in half a day is how to create security holes...
• We need to persuade the powers that be—funding agencies, departmental chairs, etc.—that this stuff deserves institutional backing. Many believe it's important; what's hard is persuading them that it's more important than things that are already in the curriculum, and that have passionate defenders. (As a chemist once said to us in frustration, "What do we take out of the undergrad curriculum to make room for this: thermodynamics or quantum mechanics?") One thing we're trying is a "driver's license" exam for the DiRAC supercomputing facility in the UK; once it's implemented, grad students who want to use it have to show they have at least basic computational competence.

We're pausing to catch our breath from mid-July to the end of August this year, during which time we'll clean up our GitHub repositories, finish the instructors' guide, and generally get ready for another round of boot camps in the fall. Before then, though, we will have some exciting news to announce, so please keep an eye on this blog—we think you'll like what you see.

Having helped at two previous boot camps (Oxford last year and UCL last month), a couple of weeks ago I had my first attempt at organising one at the University of Oxford, primarily aimed at students from the Doctoral Training Centres. Mario Antonioletti and Shoaib Sufi from the Software Sustainability Institute kindly did most of the instructing (I only ran two sessions), and Mario has already written a blog post with his perspective. Here I want to report the feedback we received from attendees and give some of my thoughts on the experience.

It took a while for registrations to get going - the vast majority signed up in the last week - but we had very few drop-outs before the event, with 30 attending the first day, and 24 still present at the end of the second. We did the traditional good point-bad point round-the-room feedback at that point, and got quite a few useful comments, shown below. I also sent round a more detailed questionnaire after the event which received 10 responses. All enjoyed the boot camp, and rated it overall good or very good. They also all felt they learned something useful, would recommend such events to others, and think universities should run them as standard.

I think we did pitch things generally at slightly too easy a level for our audience. This was particularly the case for the introductory Python session, which started quite slowly. Several responses to the questionnaire requested fewer topics but in more depth, with less time spent on the very basics. It was interesting to me that the students wanted more background to the topics taught, for instance comparing Python with other programming languages and discussing the kinds of problems for which it is particularly suited, or explaining the mental model and behind-the-scenes behaviour of Git.

There were also some differences in what attendees wanted out of the boot camp. Some were particularly keen to learn Python, and they seemed to really appreciate what we covered. Others wanted more to improve their Matlab skills, and so were disappointed that the 'Software Carpentry in Matlab' session was squeezed by earlier sessions overrunning. I think they would probably have wanted a workshop devoted to covering the general principles entirely in Matlab without any Python, too.

The location in the Zoology Department worked very well for a boot camp. It was a newly equipped suite designed for teaching computer based courses. There were thus desks with plenty of space for helpers to move behind attendees, and desktop thin clients for those who couldn't bring their own laptops (or had issues with their setup). Best of all, the instructor's screen could be projected on three walls of the room, so everyone could see a screen in front and to the side.

Most setup issues were caused by Enthought releasing Canopy between the installation instructions being written and the boot camp occurring, so those attendees who didn't already have Python were faced with an installation that didn't match the instructions. The additional setup script for Windows users was also broken by paths changing. More annoying were some issues experienced with IPython on some machines, which we didn't get to the bottom of unfortunately. Pasting with %paste didn't work for some (%cpaste seemed to be an effective work-around), and tab completion was entirely absent on some systems.

We came across a few issues (and some good ideas) that might be helpful for future boot camps to consider.

• Mario wrote 'recap slides' for the Git session, summarising where we had got to at different stages through the morning, and students found these helpful.
• Accessing BitBucket with Git led to SSL certificate issues on some machines. It's worth knowing the incantation to fix this if it happens to you (git config --global http.sslVerify 0).
• Students really found having all the material available at the start helpful.
• On that note, we should mention ipython's var.<tab> help system in the notes.
• In the testing session, we tried to get people to run Python scripts as commands by including a shebang line and setting the executable bit. Lots of people had issues following along here, and it would probably be better to run with e.g. python dna.py.
• We tried to do a longer practical exercise after the NumPy session. Much more time was needed for this! But I still think such an exercise, that includes dealing with basic Python datatypes and functions, is very valuable.
• Importing CSV files into the database is quite hard to follow along with. This is a general issue for any section that involves following button clicks and interactive dialogs. We benefitted from having a pre-canned database in the repository for anyone who gets too lost.
• Another aspect that some found difficult to follow was long bits of typing that weren't essential to understanding, for instance typing HTML sample files for the version control session.

There were a couple of other comments worth mentioning. One person wanted a topic on understanding pre-written code. Possibly going through a model solution to a longer exercise would be beneficial for this. Another person suggested splitting the workshop into half days on different topics, and letting attendees pick which they attend. I don't think this would work for normal boot camps, but with particular audiences, especially those with more background in some topics, this might be a helpful approach.

But my favourite comment of all, in one questionnaire response, was: "In my opinion the best aspect of the course was the explanations given about new concepts by the very competent computer scientists who ran the bootcamp. This was especially true for Jonathan and Aron." I think that's a good note on which to end!

On May 6th-7th, we hosted a bootcamp at Stanford University. Participants were students, post-docs and staff affiliated with the Center for Neurobiological and Cognitive Imaging and the Neuroscience Graduate Program and both units provided support for the workshop. Bob Dougherty, the research director at the CNI, helped raise a substantial portion of the funds to support the boot-camp (coffee!) and Prof. Miriam Goodman, from the Department of Molecular and Cellular Physiology, helped to get the neuroscience program on board and to secure additional funding from that program. Prof. Goodman was a participant in a SWC boot-camp at Berkeley last year and was eager to get SWC to Stanford. Instructors were Paul Ivanov, Bernhard Konrad and myself.

Considering that the students were a rather homegenous group in terms of their scientific interests (the brain!), there were a variety of backgrounds in terms of their knowledge about computers. Matching instruction to the level of knowledge (based on a pre-workshop questionnaire) was a challenge. As were installation issues that plagued the first day. A large majority of participants failed to follow the instructions provided by the Anaconda Python distribution to configure their bash shell environment. I guess that just goes to show that they needed the workshop.

The instruction followed the standard boot-camp plan for most of the two days: unix shell, version control with git, python programming, and so forth. In the last two-hour session of the work-shop we conducted a little experiment. Adapting a lesson plan from Justin Kitzes' recent UC Berkeley bootcamp, we taught a session aimed at teaching the students how to combine all the components of the previous parts (shell, git and programming) to create a reproducible research project. To make things more interesting, we used a publically available neuroscience data set. We demonstrated how one would load the data from the files comprising this data-set into an IPython notebook and performed a rather basic calculation on it, visualizing various quantities as we went along.

Demonstrating one possible work-flow, we started by working interactively in the notebook, exploring the data as we went along. We demonstrated how bits of an exploratory data analysis session can turn into function definitions. Then, we showed how you would write modules based on these function, tests for the modules, commit them as a git repo and start putting together a push-button analysis. We reached the end of all this a little bit out-of-breath and it is not obvious that everyone managed to follow through every step of the session, but I do believe that the most important point came across: reproducible research is important and, with the help of the tools we teach, eminently possible.

We used this etherpad to comminucate with students througout the workshop and to gather feedback at the end of each half-day. You can see a list of the participants at the top and the feedback strewn throughout. At the very end of the workshop, we asked them to write down their feedback for the entire workshop. Many took the time to write some thoughtful comments (for which we are grateful!). I have selected a couple of interesting points from the feedback we got. This one is quite encouraging:

great workshop. i appreciated the backround and theory as well as the more practical examples. i think you did a nice job presenting this information to students of various levels. might be helpful to make it clear what level this class is for - some people had no coding backround, others were quite experienced. maybe a quick survey in the beginning and maybe some work in groups to accodmidate the different levels. i really appreciated your humor and positive attitude:)

On the other hand, this one is a bit discouraging:

I'm very grateful that you all volunteer your time to do this. But, I think you can/should be paid for it. There's money around for such things, many of our PIs would be happy to fork up a few hundred bucks, and in general I think the culture of doing good work for free in academia is deleterious to our collective financial well-bring.+4

The '+4' at the end of that comment represents the fact that four additional people expressed their support to that point. I wonder whether this notion is due to the local hyper-capitalistic culture surrounding Stanford, and I am curious to hear what other instructors think about this. My personal response, also in line with some other aspects of the local culture, is that Software Carpentry is just one more way to create more value than we capture.

When the workshop was over, Paul, Bernhard and I took a long walk. Prof. Goodman had kindly invited us to her house in Menlo Park for post-workshop pizza and socializing. On the way over, we took the opportunity to go over the sessions one-by-one and reflect on them. For each session, each one of us in turn said what they think worked and what didn't work. It was very useuful to get feedback from the other instructors, to rehash the mistakes and the successes and to be able to imagine what we would do differently in future workshops. One of the main sentiments in the air was that we were not sure whether we had really managed to "seal the deal", to get students to actually start using the things we had shown them in their day-to-day life. Fortunately, I have the email addresses of participants. In a couple of months, I will circle back and ask them to fill another set of evaluations and (assuming anyone responds) see whether they have adopted any of it into their day-to-day research life.

Jenna Lang has posted a great wrap-up on the boot camp at UC Davis — with Python cookies!

Mario Antonioletti has posted his experiences on being a first-time instructor at our boot camp for the Oxford doctoral training centres, our second in Oxford, last week.

Hack4ac is a one-day hackathon in London, England, on July 6. Its goals are:

• Demonstrate the value of the CC-BY licence within academia. We are interested in supporting innovations around and on top of the literature.
• Reach out to academics who are keen to learn or improve their programming skills to better their research. We're especially interested in academics who have never coded before.

It looks like fun—if you're in the area and interested in getting involved, they'd be happy to have you there.

Vijee Venkatraman has written a good article for Science Careers titled "When All Science Becomes Data Science", which mentions Software Carpentry.

Software Carpentry's mission is to help scientists teach other scientists how to be better programmers. If we want to do that successfully, we need to be scientists ourselves. In particular, we need to base what we teach on evidence, not anecdotes or personal preferences.

For example: we taught Git at the Toronto boot camp last week, and once again I think our learners would have absorbed more if we'd taught Subversion. Why? Well, take a look at this diagram by Oliver Steele (which I found on this page written by Nick Quaranto):

Four locations are in play, and eight different commands are used to move information around or compare what's in one place to another. Now look at the corresponding diagram for Subversion:

There are only three locations for people to keep track of, and the basic workflow involves only four commands. If you believe complexity is partially multiplicative (because people have to keep track of the interactions between things, as well as the things themselves), Git is at least twice as complex for people to understand. Slicing it another way, there are more opportunities for people to do the wrong thing with Git, and more they have to understand to undo it. And that's before we introduce branches...

But that's just my point of view, and the fact that I can wrap a plausible story around it doesn't make it true. Matt Davis and others believe they've been successful with Git in front of the same kinds of people. We need to find out who's right, so our major goal for the next three months is to poll and interview boot camp attendees to find out who was taught what, who tried what, who's still using what, and why. Caitlyn Pickens will be leading this effort, and many of you will get email from her in the next few weeks. When it arrives, please take a few minutes to tell us what you're actually doing, even if it's not what we told you to do—especially if it's not, because that'll tell us what we need to fix.

The hosts of our February boot camp at the AMOS conference in Melbourne have collected some more detailed feedback from participants. I'm pleased that two thirds thought the content was just right, and even more pleased that 83% thought version control "must be taught".

How useful did you find the online software installation instructions?

How did you feel about the schedule?

The lunch and tea breaks were:

What do you think is the ideal length of time for a boot camp?

Rate the following topics in terms of how important you think they are for a weather/climate science audience.

In general, you found the content:

"Online office hours" is a service provided by Software Carpentry where you can get online assistance over a 2 hour period once a week. Which item best describes your current engagement with that service?

If a boot camp (of similar content) was held in conjunction with the AMOS conference in Hobart next February, do you think you would attend (i.e. as a refresher)?