Blog

We're very excited to announce the launch of the Mozilla Science Lab, a new initiative that will help researchers around the world use the open web to shape science's future. The Science Lab will foster dialog between the open web community and scientific researchers. Together they'll share ideas, tools, and best practices for using next-generation web solutions to solve real problems in science, and explore ways to make research more agile and collaborative.

The project will be lead by Kaitlin Thaney, a long-time open science advocate. Kaitlin helped found and manage the science program at Creative Commons, and previously worked at Digital Science, a tech company focused on research tools and incubation for science startups. She also advises policymakers on digital infrastructure, data-intensive science, and education to make scientific research more collaborative, open, and reproducible.

For more information, please follow the Science Lab and Kaitlin Thaney on Twitter, or see the announcement from Mark Surman the director of the Mozilla Foundation, or this post by Kaitlin.

We're pretty excited :-)

Earlier this week, Software Carpentry had its first online lab meeting since October 2012. In attendance were:

We covered a fair bit of ground in 90 minutes; the highlights are below, and comments from both attendees and everyone else would be very welcome (either here or on the discuss mailing list).

Making boot camps more useful:

1. Clearly distinguish boot camps for complete beginners from those for people who already have some coding skills.
2. Do not publicize the boot camp until the curriculum has been specified so that learners can make an informed choice about signing up.
3. Provide a consistent pre-assessment to help people make this decision before they sign up (and to give instructors a clear idea of who they're going to be teaching).
4. Encourage people to teach boot camps spread out over several days or weeks when instructors and learners are co-located.

Sustainability (see this post for background):

1. Poll hosts of previous boot camps to find out what their actual costs were, and whether they would have been willing/able to pay a $1000-$1500 administration fee per boot camp.
2. Depending on the results of that poll, start charging those who can pay an administration fee for setting up boot camps.
3. Explore the PLOS model of different fees and waivers for different countries.
4. Pursue corporate sponsorship (either on a per-bootcamp basis, or as general donations to the cause).
5. Try once again to get funding through existing science training programs.
6. Encourage qualified instructors to teach Software Carpentry commercially, for whatever fees they can charge. (We will not be involved in fee negotiations, and will not "tax" these activities, but would appreciate instructors talking with us before approaching clients so that we're not tripping over each other.)
7. Create a new mailing list for people interested in discussing this. (If you'd like to be added please mail me.)

Consistency:

1. To use our name and logo:
   • At least one qualified instructor.
   • At least two full days (though this may be spread out over a longer period).
   • Cover version control, testing, task automation, and modular program development.
   • Most attendees to date work in science, engineering, medicine, and related fields (we're not teaching web development to salespeople). We are already extending our reach to people in other data-intensive disciplines, but for now at least, the digital humanities are out of reach: we simply don't have enough instructors (or enough experience).
2. To be an instructor:
   • Complete our online training (which includes teaching at least once with a more experienced instructor),
     or
   • Demonstrate extensive relevant teaching experience (and teach at least once with a qualified instructor).
3. These requirements will only affect upcoming boot camps and instructors, not those already scheduled or badged.

Simplifying our shared content, web presence, and workflow.

1. Designate a lead instructor for each boot camp.
2. Streamline the reimbursement process. (Ideally, we want host sites to reimburse instructors directly, but we realize that we'll often be involved, particularly when people are teaching several boot camps in one trip.)
3. Make it much easier for people who are not yet Git experts to update boot camp web pages, contribute teaching materials, etc. We are discussing this on the gits mailing list; please join us there.

Notes:

• Some people proposed having a warm-up day for complete beginners, followed by a regular two-day boot camp that would appeal to people who already have some skills. We'll need to experiment with this to see how well it works.
• There was disagreement over whether testing should remain a required topic: some instructors barely touch on it, and others disagreed over what it ought to mean for day-to-day scientific computing. As Software Carpentry's CDFL¹, I ruled that it is required; however, we need to integrate into the other material rather than teaching it as a standalone topic.

[1] Curmudgeonly Dictator for Life.

Last month, we held two Software Carpentry workshops back-to-back in Durham, North Carolina, one at the National Evolutionary Synthesis Center and a second organized by Cliburn Chan for the biostats group at Duke just three days later. These workshops were taught by Jenny Bryan, Elliott Hauser, and myself with Karen Cranston teaching as well at NESCent. The workshops were both somewhat atypical: NESCent was a smaller workshop (only 11 participants) who work together and share an interest in evolutionary biology, and at both NESCent and Duke we presented in R rather than Python. Overall, both workshops were very successful, and I learned a lot, both about R and teaching a Software Carpentry workshop.

In addition to substituting R for Python, we altered the schedule from what I've seen in previous workshops, teaching R for the entire first day and everything else (shell, git, and Make) on day two. While this meant that Jenny had to present the entire day on day one, her material and instruction style easily held everyone's interest for the whole day. Jenny went at a good pace, moving from basic language essentials and how to interact with RStudio to more complex topics such as data aggregation using the plyr package and producing literate documents using knitr. The R sections, particularly the use of plyr, were consistent crowd favorites. I think the way we rearranged the schedule worked out very well—we avoided confusion by not switching between environments, and using only RStudio on day one allowed us to avoid installation issues. Anyone who had had trouble installing the prerequisites either stayed after on day one or came early on day two for help. We ended up with no setup issues at NESCent and a fair number at Duke (possibly due to inadequate instructions on our boot camp page) but they were easily handled.

We were able, perhaps serendipitously, to present our material in a logically cohesive way that kept students engaged. Instead of every instructor bringing their own material, we used the same files for all sections. The code we wrote during day one was placed under version control on day two, and we wrote a Makefile together to automate our data analysis pipeline and produce an HTML document with code and figures. At NESCent, this wasn't planned ahead of time; I noticed that Jenny had designed her scripts in such a way that they were great candidates to incorporate into my Make tutorial, and so together we wrote one. There was clear interest in Make, so we built on this idea and replicated it at Duke with more preparation.

The number of people attending the NESCent workshop was low, which was in some ways both a pro and a con. On the plus side, we were able to pay more individual attention and Karen led a discussion on data management that was highly relevant to those attending. However, we seemed to run out of steam during the afternoon on day two, where we had planned to address whatever topics participants felt would be helpful but found a lack of enthusiastic suggestions. We corrected this at Duke by being prepared for a more structured presentation. We also found for both the shell and version control sections that our presentation needed to cater to specific tasks that a researcher just learning these skills would use and avoid more complicated topics such as forking and pull requests.

Teaching two workshops back-to-back gave us a chance to refine our approach and materials, and Jenny is looking into how best to incorporate the materials we designed into Software Carpentry for future R workshops to utilize.

NESCent feedback

Duke feedback

Alex Viana and myself recently wrapped up a bootcamp at the National Center for Atmosphere Research (NCAR) in Boulder, CO. It was a lesson in the advantages of being light on your feet when it comes to teaching for Software Carpentry. Our normal audience tends to be scientists with varying degrees of familiarity with the tools that we teach. This time we taught about 20 people all of whom were experience software engineers, some of whom had probably been programming longer than Alex or I had been alive. We had planned on starting the morning with some shell and then moving into Git. A quick survey of the room found that everyone was well versed in the shell so we abandoned that lesson all together and went straight to teaching Git.

Here was where things got interesting. I thought we'd fly through our material because everyone was so experienced. Instead though it almost seemed that we moved more slowly because not only were we teaching Git, but also how git was different than SVN and CVS (other version control systems for the uninitiated). Later in the day we also got into a deep conversation about what types of workflow Git could be used for compared to SVN and CVS.

We had a similar experience the next day teaching Python. Instead of just churning through it we spent a lot of our time talking about how Python was different than other languages the participants programmed in like Java. It turned out that teaching such an advanced crowd meant that we had a group that knew what questions to ask. Topics that we normally gloss over (what is an iterator anyway?) we had to really pick apart because of the knowledge in the room.

As instructors there were two really important lessons for us. The first is never assume that you're the most knowledgeable person in the room. The corollary to that is that just because you aren't doesn't mean you don't have something to teach. We even had a couple of really skilled Python programmers who were happy to be there just to see how they could become better teachers. At first we were a bit intimidated by the depth of knowledge, but really it just forced us to up our game as instructors and I think we're better for it.

The second lesson was that you can never forget that a bootcamp needs to be adaptable. Alex and I were challenged to create curriculum on the spot when we skipped over our 2 hours of the shell. We had to keep in mind that we were there to teach not just stand and deliver a canned lesson even if participants already knew it all. And that's the really great thing about Software Carpentry all around. All the instructors are creative and smart people who are dedicated to being good teachers and don't shy away from these kinds of unexpected challenges. The NCAR bootcamp was definitely my most challenging, but it also offered one of the best opportunities for growth as an instructor.

On 21st August 2013, we'll be attending INTECOL13 in London's Docklands. The 11th International Congress of Ecology, hosted by the British Ecological Society and INTECOL, is one of the world's premier conference for ecologists. At the invitation of Matthew Smith of the BES Computational Ecology Specialist Interest Group and Microsoft Research, Software Carpentry and The Software Sustainability Institute have a talk and a session.

Steve Crouch will present a talk on behalf of Greg Wilson, on "better computational practices for research". This session will highlight the problems faced by researchers who develop software, and ecologists in particular, including the tractability, development and reproducibility of ecological models, and discuss possible solutions.

After runs at Effective Scientific Programming, Digital Social Research and Dev8D, Mike Jackson will chair another discussion on "what makes good code good". This session will explore what ecologists view as good code, how ecologists learn about good code, the blockers to producing good code and how these can be overcome, by research leaders, institutions, funders, and ourselves.

We look forward to reporting on our experiences.

It's been eighteen months since we started running two-day boot camps. We've grown rapidly:

Cumulative Number of Boot Camps	Cumulative Enrolment

Now we need to start thinking about how to sustain that growth in the long term. To start, here are some stats on how things have been funded.

First, we've experimented with several different ways to fund travel costs, and several different ways of charging attendees:

Here's how often we've used each combination:

One fact that doesn't show up in this table is that the number of grant-funded boot camps has gone from 16 in 2012 to 2 in 2013. Even if we include two mixed-model boot camps this year in the latter count, we're clearly much closer to boot camps being self-sustaining.

But being closer isn't the same as being there. We're spending about $150K/year in grant money on salaries and overheads to keep this whole show going. At two boot camps a week (which we think we can reach by this time next year), we'd need about $1500 per boot camp to cover those costs. If we stick to an average of 40 learners per boot camp, we'd have to charge a bit less than $40 per person. The problem is, we often can't do this: if we charge for training—even a refundable deposit—then some institutions will charge us for use of their space, and only charging those who don't have such a rule seems unfair. An alternative is to ask for sponsorship on top of instructors' costs, rather than tying it directly to enrolment.

Setting that issue aside for a moment, though, I'm more certain than ever that this training is cost-effective:

1. The average attendee is a graduate student who costs about $50,000/year in salary and overheads (mostly overheads). If she works 40 hours a week, 48 weeks a year, that means her time is worth about $26/hr.
2. Since January 2012, we've run about 75 boot camps for about 2500 people. If each boot camp is two eight-hour days, that works out to $1,040,000 worth of people's time.
3. Instructors' travel costs have averaged about $2000 per boot camp, but we'll round up to $2500 to account for local organizers' time, coffee, etc. That gives us another $187,500.
4. The $150K/year burn rate quoted earlier for salaries and overheads gives us another $225,000, for a total of $1,452,500.
5. But that's only $580 per learner, so at $26/hr, we only have to save people about 22 hours each for this to be worthwhile.
6. Looking at it another way, if instead we save them an hour a week over five years, that's $6240 worth of their time, which is 11 times the up-front investment.

Measured that way, I'd say we're doing more than OK.

What started out as an unexpected proposal from Greg became very much alive on May 2nd, as 40 physicists poured into the classroom at VU University Amsterdam and started up their laptops. After introducing the teachers (Stefano Cozzini and Justin Ely) and the helpers, we were off: the very first Software Carpentry boot camp in the Netherlands!

In the months leading up to the boot camp, I had been surprised at the sense of timeliness the boot camp generated. People here clearly see the need for computational training, most importantly young scientists themselves—available tickets were mostly gone within a week.

The program of the workshop had been announced as being geared towards teaching you "how to build better code in less time". It included the usual Software Carpentry topics: the UNIX shell, Python programming, version control (Subversion, by popular request), and some data management and matplotlib.

During the two days of the boot camp, Justin and Stefano did a great job at teaching these topics, mixing interactive sessions with some more theoretical material on software lifecycles. Especially the session on Subversion was a big hit: people made over 80 commits to our shared repository within an hour, and seemed to grasp the material well. (Today, an attendee told me he's actually using it for his work now).

Some other things that worked well included:

• The little trick with green/red sticky notes to signal helpers;
• Frequent, short breaks;
• The VirtualBox VM: this pre-empted a lot of setup issues (we encountered very few of those).

And yet, I'm not sure the boot camp was a definite success overall. Looking back, and reading the feedback, we hit two snags.

The first will be familiar to Software Carpentry regulars: we had a large, inhomogeneous group. This made it very hard to teach everyone something new, while at the same time not losing anyone. Especially while teaching Python, we stuck too much to tutorial-level topics. This didn't leave enough time for more advanced topics such as program structuring, debugging, or object-oriented programming.

Secondly, and more importantly, I'm not sure we kept our promise of teaching "how to build better code in less time". Sure, judging from the feedback, some people clearly got useful skills out of the workshop. But quotes like this one…

The bootcamp was advertised as saving time writing code. I really wanted to learn that, and I don't think I did see anything saving time.

…and this one…

I went looking for strategy in software building, learning how to make a plan and follow it together with other people. Sadly the course stayed in a much more introductory stage to particular tools.

…show that this was not enough.

Could we have done better? I think we could have, if we'd put more effort into teaching materials in a useful context. This was one of the reasons why the version control topic was a success: people went through real-life scenarios, got their hands dirty, and learnt something useful that way. Doing something similar for Python programming—letting people work on a real program, while learning about structure, documentation, refactoring, etc.—would have made a difference. Many attendees are longing for such realistic examples.

And, of course, if you're organizing a boot camp: think twice about the type of audience you're inviting. A generic mix of everything from novice to expert will make teaching the boot camp more difficult. That's certainly something I'm going to keep in mind for next time.

I have recently organised a boot camp at the University of Southampton, as part of my Software Sustainability Institute fellowship. I hadn't actually attended a boot camp before (I was meant to, but unfortunately had to cancel) so it was a little nerve-wracking doing all of the organising - it all worked out well in the end though.

The boot camp was primarily for students at the Institute for Complex Systems Simulation, from which we got around 35 students, with 15 students from the university's Computational Modelling Group making up a grand total of 50 attendees. As the course was primarily designed for the ICSS, I did a very early pre-questionnaire asking what topics the ICSS students would like to cover - which was very much appreciated - and this early discussion got registrations off to a good start. In fact, we had around 20 people on the waiting list at one point, which allowed us to easily fill up places when people dropped out. In the end we had 48 people registered on the day, of whom 42 attended.

There were some changes in the team of instructors during the planning of the boot camp, as one instructor dropped out part way through the planning process, but we still managed to keep going with the same topics we had planned, and were instructed by Nelle Varoquaux and James Morrison. The ICSS at Southampton kindly paid expenses for the instructors, and provided us with very tasty buffet lunches as well as coffee/cakes for the morning and afternoon breaks.

At the end of the final day we did the traditional good-point, bad-point questions around the room and got the useful comments below:

From the post-event questionnaire, 100% of respondants found the boot camp enjoyable, useful and will apply some of the techniques in their work.

The room we managed to book for the boot camp (85 / 2207 for anyone reading this in future before another boot camp at Southampton) worked very well. It was large enough to fit all of the attendees in, was air conditioned, had plenty of space for helpers to move around, and had space outside for people to mill around during the lunch and coffee breaks. We deliberately re-arranged the room to have a large central aisle with tables either side, so that the helpers only had a maximum of four people to squeeze past to help someone, and this was very effective. The only improvement to the room would have been having two projectors - we did bring a spare projector and use it, but it was so dim that we decided we couldn't really use it for important things (hence putting the Etherpad on there).

As with all bootcamps, we struggled with a wide mix of students coming to the course - some of which had very little (if any) experience with Python. However, we managed to keep most people with us, and I don't think anyone didn't turn up for the second day. We had a lovely comment that "I am really pleased with myself that I stayed until 5pm and am coming back tomorrow - so that is good news, thanks!". In fact, that comment came through the brilliant red/green post-it note feedback mechanism. We used these post-it notes in the standard 'notification'-type way (where students stick a red post-it on their laptop when they have problems, and a green when they have done the exercise), but we also got students to write on the post-it notes before each break in a session, listing a good thing that they really understood on the green post-it, and something they were struggling with on the red post-it note. This was really useful for the instructors, and we were able to tailor the subsequent part of these sessions to answer the specific questions that were raised.

In terms of specific things that might help future boot-camp organisers and instructors:

• There were significant problems with the setup and installation procedures. Many people didn't do this before-hand, or tried to do it, found it difficult and then gave up. This caused significant problems in the first session, with some people managing to get going with the shell session, and others still struggling with the setup. There are various things that can be done about this:
  • Ensure there is double the amount of time you think you'll need available at the start of the first day for setup issues.
  • Email people the day before asking them to come early if they have had setup problems.
  • Make sure you have a VM image on a pen-drive for students who have issues (I forgot to do this...).
  • More generally: I think we need to significantly re-write the installation instructions on the Software Carpentry webpages. A number of students said that they tried to follow them but that they were very confusing and made very little sense. One student commented that we really need to do a 'Step by Step Installation for Dummies' guide which goes from nothing to everything, and shows how to check if things are installed. I agree, although this will be difficult to make applicable for every system.
  • I felt a bit embarassed that we dropped James (who was instructing for the first time) into these setup problems on the first day, I think the first session should ideally be run by an experienced instructor who will be more familiar with dealing with these problems.
  • Students were asked to clone the git branch for the boot camp at the beginning of the first session - before they had been taught git! They found this very difficult, and many people didn't get it properly downloaded for a while. The other problem is that doing a standard 'git clone URL' will download all of the data for that repository (including all branches, all of the history etc) - which took a LONG time when 50 people were doing it at the same time. One student very reasonably asked why we don't provide the boot-camp material as a simple zip file.
• The red/green post-it system for feedback at the breaks in each session was wonderful, and should be used more widely.
• It is amazing how much a good food plus coffee and cakes can help...although it is very important to make sure dietary requirements are taken care of (even if the catering organisation said they would provide it ok, they didn't).
• We tried to make a more flexible final session by pointing students to the Software Carpentry resources, and the resources from the boot camp, and asking them to work on whatever they wanted and have lots of discussions about issues and general questions with the instructors. However, many students took this as a cue to leave (which I hadn't expected!) and so I had to hurridly give my final presentation.
• I think a number of students found the final presentation which brought it all together quite useful - but I altered the 'standard' Software Carpentry one quite a bit, to add in ways to combine all of what we'd learnt (eg. shell scripts as git hooks to run automated tests). This new presentation is in the 2013-06-southampton branch of the boot-camps repository.
• Nelle's reproducible research session was very much appreciated by people, particularly the use of makefiles for reproducible research.
• A number of students found the testing session rather frustrating for a number of reasons including: too much Python coding, more time spent on it than they thought it needed, not understanding the point for people who don't write libraries etc.
• One major suggestion from a number of people was to make all of the examples inter-linked, and to present the boot-camp as one big story, showing how you'd use these various tools and techniques to do 'good reproducible computational science'. This is a lovely idea, but I can imagine it would be very difficult - however, I may blog about it in future.

I think the structure of the boot camp worked very well, particularly as we had a lot of time allocated for each session, so we could go into more depth but also not worry too much about time. However, the real stars of the show were the instructors - and they deserve huge thanks. I think it is appropriate to finish with a comment from one of the students on one of their green post-it notes:

Thank you for selecting such knowledgable people to instruct this course!

This blog post is actually a bit overdue as the first time I instructed at a Software Carpentry bootcamp was over a month ago in Manchester. However, as now I had an opportunity to teach twice (recently in Krakow), this gives me more credit to share my experience of taking a step up from being a helper to being an instructor.

I had helped at bootcamps in the UK so I knew how the materials were taught and what they covered. Nevertheless, it still took me a lot of time to go through the modules I was supposed to teach making sure that I could seamlessly not only follow all topics, examples, and exercises but also present them in a clear manner. As a helper most of what I had to do was to solve ad-hoc problems reported by the bootcamps attendees. As an instructor I had to make sure that everything I was saying was consistent with what I said before and logically led to the next step.

I also learnt first hand that addressing a diversified audience is always challenging. In both Manchester and Krakow I experienced the difficulties of teaching a group who (according to the pre-questionnaire) varied from those who, for example, had never used Python at all to those who "strongly agreed" that they can write a Python program. This meant that I was going to either lose half of the group or make the other half terribly bored. I thought that the former was worse. Software Carpentry is supposed to be improving researchers' computing skills and not discouraging them even more from the world of programming, testing, and using version control. I figured that those who were already quite familiar with the material would catch up with their Facebook in the worst case, and refresh their memory and maybe learn a few details they didn't know in the best case. Pitching at the right level is difficult.

About a year ago I witnessed a situation (not at a SWC bootcamp) when an experienced developer was trying to explain to a complete novice the mysteries of Git. To cut a long story short, it didn't go well—at the end the novice was still clueless, but what's worse, the developer also seamed somewhat lost in his explanations. As I was supposed to teach Git, this memory became quite fresh in my mind. Fortunately, I was able to observe as a helper how Mike Jackson taught Git at previous Software Carpentry bootcamps and I knew that teaching distributed version control can be done effectively without confusing the students and yourself. I used practically the same materials which Mike presented introducing only small tweaks and followed Mike's teaching approach.

On more practical note, I learnt it the hard way that teaching a two-hour block standing rather than sitting affects my typing skills pretty badly. After about 15 minutes my back and wrists were aching, I had to look at my laptop monitor from a very weird angle and the number of typos which I made was growing exponentially. It all could have been avoided if I got myself a chair to sit on.

Instructing at Software Carpentry bootcamps is probably a different experience for everyone. Although, as I had a chance to talk to two other people who recently taught for the first time, there was one thing which we agreed on: allocate sufficient time to prepare for the bootcamp. Don't assume that it'll be enough just to flick through the materials and copy them over to your bootcamp branch. If you are an experienced instructor, this may work. But if you've only been a helper until now, you'll need to put more effort. But the payoff when you see the learners going "Ah, now I get it", is well worth it.

The Software Sustainability Institute's Simon Hettrick writes:

Intellectual Property (IP) is a thorny issue in academia, generally because people don't know who owns what. There's a lot of hearsay, rumour and—frankly—misunderstanding about IP ownership, so we've decided to run an experiment. And we're going to need some guinea pigs…
…
To get a better understanding of how academia deals with IP ownership, we're running The Great Licenceathon. This will see people at different organisations working out who owns their IP. They'll then blog about their experiences and we'll pick through those blogs to see whether there's some common lessons we can learn.
…
If you want to know who owns your IP, and you have time to ask a few questions at your organisation, then please get in touch.

I think this information will be extremely useful, not least because we're thinking about adding discussion of licensing and open access to the "day 3" material for our boot camps. Please check out the entire blog post, and please do get in touch with Simon.