Blog

(Posted 2013-06-19 by Greg Wilson)

Most musicians can play along with a twelve-bar blues once they know what the key and tempo are. Many kinds of scientific work are equally well structured: the results aren't predictable—it wouldn't be research if they were—but the equipment setup, sample preparation, note-taking, statistics, and write-up are (mostly) structured in ways that other scientists are familiar with. This lets them pick up each other's projects more quickly; it also gives scientists more time to do what's unique about a particular project because they don't have to spend time thinking about the things that aren't.

Open science—by which I mean all the new ways of doing science that the web has inspired—isn't there yet. Many people have tools and techniques that work well for them, but every setup is one-of-a-kind, and everyone has had to assemble the pieces for themselves. I think we could do better, and the thing I think about when I think that is Ruby on Rails.

A lot of Ruby on Rails' original simplicity is now hidden under a mass of extensions and auxiliary tools, but when it first appeared in 2004, it was a minor revolution. One reason was the "create a blog in 15 minutes" screencast that showed people just how easy simple things could be. Another was its emphasis on convention over configuration: instead of letting web developers do things however they wanted, it said, "Here's where stuff is and here's how it works." Once a competent developer knew what the application was supposed to do, she could (almost) immediately start building the things that were specific to it, and other competent developers could join in (more) easily. What's more, Rails' predictable structure and workflow made it easier for newcomers to adopt—certainly easier than contemporary competitors, which all-too-often required novices to make key decisions before they had the knowledge or experience to do so well.

So what's the equivalent for science? What would be in a combined template/tool/framework/worldview for small- to medium-sized scientific projects that would let scientists do science our way with near-zero startup overhead? (Note that I'm not asking, "What web programming framework should they use?") A few things I want are:

1. Data organized in a particular way. I'd use William Stafford Noble's "A Quick Guide to Organizing Computational Biology Projects" as a starting point, though it's obviously not right for all cases. I also realize that projects using Data of Unusual Size will probably store metadata and slices locally, rather than entire data sets, but that's no worse than the diversity of data, images, and auxiliary files stored by some small web apps.
2. Scripts too. The little bits of code used to do "last lap" statistics, create graphs, and so on should be in a predictable place, and be invokeable in uniform ways.
3. Everything automatically accessible on the web, for the long term. As soon as I create a project, other scientists should be able to query my data—not just download it, but query it. OData and GData are a step toward a solution for data; the scientific equivalent of rails new project should automatically generate the wrappers needed to pull a project's data and metadata. Everything else should automatically be accessible too: every paper, figure, or table should have a DOI and be searchable and fetchable. And note that "accessible" doesn't just mean "the bits are available": without correct documentation of formats and semantics—which most scientists never quite get around to writing by hand—data rusts almost as quickly as code.
4. Standard commands to tie it all together. I mentioned rails new project a moment ago; Rails, Django, and other frameworks of the same ilk create a command application for each project so that people can add new features in a reliable, findable way with a few keystrokes. Science projects should have something similar for adding new data sets, producing new results, etc.
5. Pluggable. Rails didn't just put the application code and data in particular places; it also provided slots for adding new management commands and analysis tools.
6. Forkable and mergeable. Whatever Rails-like system scientists use must play nicely with version control, because as I wrote a couple of weeks ago, future scientists won't submit, publish, and download papers: they will fork and merge projects.
7. Federated. The web lets us access information stored anywhere on the planet; ironically, our response so far has been to centralize like never before. Wikipedia, Facebook, and GitHub are effectively all single points of failure: one court order, change of policy, or (as we recently discovered with Mendeley) ideologically incompatible acquisition can have wide-reaching impact. We know how to distribute information so that it can survive the disappearance of its initial host—BitTorrent is just one example. I hope that whatever else "Science on Rails" is, it will be robustly federated from the ground up.

What else would you add to this list? What would you want to see in a framework for 4×8 projects (i.e., ones where four people work together for eight months to produce a result)? What pieces of the solution are already out there, waiting to be integrated?

(Posted 2013-06-18 by Terri Yu)

Background

A few weeks ago, I helped out with the Software Carpentry boot camp at UMass Amherst. It was my first time being a helper. Previously, my exposure to Software Carpentry was the MOOC style online Spring 2011 class, which I took during physics graduate school. I was among the 5-10% of students to finish the course. The low completion rate was one of the reasons Software Carpentry switched to doing live, on-site boot camps.

A couple years later, I got in touch with Greg Wilson and he mentioned that there was a boot camp coming up in my area. He invited me to join as a helper, I agreed, and I suddenly found myself on several mailing lists. I had no idea what to expect, but the boot camp turned out to be quite fun and I met many interesting people—including a core Python developer.

Post boot camp, I had many thoughts about the logistics and teaching. The staff invited me to write these up in a blog post.

What went well

For the UMass Amherst boot camp, we had two instructors: Tracy Teal and Erik Bray. They have very different backgrounds—Tracy is a biological scientist, and Erik is a software engineer who works with physical scientists. I felt this diversity was a real asset to the instruction because each instructor's teaching had a slightly different flavor. Tracy emphasized the practical side of things more, whereas Erik would sometimes mention nuances that are intellectually interesting to a CS person. Hopefully, that meant that there was something for everyone: practical stuff for the pragmatists, nuances for the computing nerds, variety for those who like variety.

There was good energy in the room and people asked a lot of questions. I liked how there were multiple ways to get help—ask the teacher in front of the entire class, put a pink sticky on your laptop or raise your hand to get the attention of a helper, or ask questions on the Etherpad.

The Etherpad was pretty cool, in my opinion, though I'm curious what the students thought. Sometimes advanced questions were asked on the Etherpad, which was nice because the whole staff could see the questions and someone who understood that particular topic could chime in, and also because people could ask their expert or off-topic questions without intimidating beginners or disrupting the flow of the main instructor. For example, someone wanted to know how to setup remote desktop software when the instructor was discussing SSH. I even asked a couple questions on the Etherpad.

I experienced IPython notebooks for the first time, and that was neat. I've used Mathematica frequently for theoretical physics research, and it was nice to see a Mathematica notebook interface in Python. My guess is that a student might have seen Python before, but they probably didn't know about IPython notebook. So even if they already knew how to program in Python, at least the IPython notebook was novel.

Technical difficulties

I didn't expect to run into difficulties with software, since I was a helper and supposedly "good" at this. Unfortunately, I didn't test all my software beforehand and that mistake haunted me. The "cool", novel technologies I just mentioned gave me quite a few headaches.

Not everyone's IPython install worked, including mine. I discovered that my version of IPython was 0.12 and not 0.13, since I had used apt-get to install IPython. The Ubuntu IPython repository was not the latest. I had also installed Anaconda just in case, but my path wasn't finding it, so I had to explicitly call the Anaconda IPython which was 0.13. I was frantically trying to troubleshoot this problem during the first 15 minutes of Day 2. I wish we had made everyone test their IPython install to make sure it was version >= 0.13, and that it properly loaded notebooks. As far as I know, there is no explicit test for IPython right now in the Software Carpentry materials.

Even the Etherpad, which seems like a fairly foolproof thing (how can you mess up a web GUI tool?), was not without its problems. As it turned out, my major responsibility at the boot camp was to take Etherpad notes on what the instructors were saying/doing. I occasionally deleted some Etherpad text by accident and undo didn't restore everything. I tried rewinding the history, but it wasn't ideal. Later, I figured out that there is a save button and I tried to save more often.

It looks bad if the helpers can't even use the software properly. I suggest in the future, to remind the helpers and instructors to test all of their software, especially tools they are unfamiliar with (e.g. IPython). And tell whichever helper is taking Etherpad notes, to use the save button often!

Managing volunteers

Overall, I think the staff could have done a better job managing the helpers and integrating them with the instructors. Pre boot camp, I often had no idea what was going on. The chatter on the mailing list was almost exclusively instructors discussing logistical matters with each other, which made me, as a helper, feel left out. I had no idea how the waiting list worked (for the UMass boot camp, the number of spots was restricted) and I had no idea what I would be doing at the boot camp, until I explicitly asked about these things. The organization could definitely have been improved.

During the boot camp, I didn't have as much interaction with the instructors and other helpers as I would have liked. Any interaction that happened had to be initiated by myself—for example, I hung around after the end of the sessions to chat. Some of this was understandable since everyone was very busy and tired. However, it would be nice if the instructors and helpers could hang out at some point before or after the boot camp—have a meal, coffee, etc. Software Carpentry only works if there are volunteers, so increasing the effort to make helpers feel welcome increases the probability that helpers will volunteer for future boot camps and spread good words about the organization.

Teaching Git

I wasn't sure that people were understanding the material on Git. Part of the problem is that Git is simply a tough subject to learn and teach, especially in such a limited time frame. The lesson on Git started off with talking about SSH (because SSH keys are needed to interact with the GitHub repo) and then 5+ minutes of talking about abstract concepts in Git without any diagrams, slides, or visuals. I was really confused about why we were spending so much time talking about SSH and not talking about Git. I also think diagrams of the different repositories, commits, etc. could be introduced earlier.

On the plus side, the exercise of everyone pushing and pulling on the GitHub repo at the same time was pretty fun. It definitely spiced up what could be a boring afternoon of version control. More importantly, the exercise was both fun and educational.

Big picture and why scientists should care about software

I've always thought that motivation is the most important part of giving a scientific talk, and the same should apply to boot camps. That said, I'm not convinced we did the best job motivating our educational material at UMass. I think it would be helpful to set aside a significant period of time to talk about why open and reproducible research is important, both at the beginning and the end of the boot camp. I'm not sure that everyone even knows what open source software is, or the important role it's played in the history of mankind. I can see that one wouldn't want to overly evangelize particular ideas and tools, but in my opinion, open source and reproducible research is worth a tiny bit of evangelizing.

In fact, this discussion could be fun and interesting. One could show a few examples of scandals or big-time retractions and even ask the participants to share their experiences. The recent headlines about the Excel error in austerity research comes to mind (testing anyone?) The reason I got interested in Software Carpentry in the first place was because of a Nature article about prominent scientific papers being retracted due to poor software. I found that to be an attention-grabbing, compelling argument at the time. Show people both the carrot (you'll save so much time with software skills, people are more likely to cite you because your software works!) and the stick (career ending retraction of results, wasting your lab's time reproducing results that are wrong).

I know Software Carpentry tries to avoid PowerPoint style presentations, but I'm in favor of some slides on open and reproducible research. As an example, I like the wrap-up presentation that Robin Wilson wrote for the recent Southampton boot camp. Robin reported that the presentation went over pretty well.

Real-time assessment

I wonder if anyone has tried in-class assessment at a boot camp, similar to the trend of using "clickers" in introductory science classes. Angel Brady at Princeton wrote an excellent post summarizing the ways in which you can do "virtual clickers". I really have no idea how this would work for a Software Carpentry boot camp, but I'm just throwing the idea out there. I am concerned that in-class assessment might be an extra layer of complication in a boot camp that is already stuffed with different technologies and material.

Following up after the boot camp

Followup is important because it's so easy to be busy with other stuff after the boot camp is over and forget the material or lose energy and interest in using it.

Office hours

Currently, Software Carpentry holds office hours after the boot camps. This is a great idea, though I'm not sure how many people actually taken advantage of it, in practice. Moreover, the office hours are pretty limited, no more than once a week. Apparently, the last office hours were held in April, the UMass boot camp was held in May, and no new office hours have yet been announced on the webpage. I think the office hours could be supplemented by other followup methods.

Web tutorials

During the boot camp, I noticed a lot of interest on the Etherpad about supplemental material. People were interested in tutorials for Python, Git, even vi/vim. Instead of staff informally writing in some links on the Etherpad, maybe this could be done in a more organized fashion. There might be a semi-official, curated list of internet resources. Online software education has become a small industry in itself, and there are lots of great free web tutorials at places like Code School and Git Immersion.

IRC channels

One little known resource I want to emphasize is IRC channels. They are certainly not mainstream; I've always been tech savvy and barely knew of their existence. I thought they were a sketchy place to meet strangers. It wasn't until I started doing open source software development that I used IRC and discovered what an outstanding resource it is. I was able to ask questions on the #debian and #ubuntu channels and immediately receive incredibly helpful feedback on how to solve install problems. I notice there are IRC channels for R, Python, SciPy, and Git. Chatzilla is a Firefox add-on and a nice IRC client that people could start with. If people are intimidated about asking questions on IRC, someone more experienced with IRC can tag along as a buddy. I would be happy to volunteer.
In fact, Software Carpentry has its own IRC channel #swcarpentry on Freenode, and I suspect that most of the instructors don't even know about it. At the moment, I'm working on open source software and am logged onto IRC all the time. I'm happy to be available on #swcarpentry. If Software Carpentry recruited more open source developers as helpers, they might be willing to staff the channel. IRC channels have huge potential as a resource for Software Carpentry and I'd like to see them emphasized more at boot camps. What could be better than a place online where you can interact with people real-time and get immediate help at almost any reasonable hour?

Meetup groups

Further, if you live in the right area, there might be an amazing software related Meetup group near you. Bay Area R Users Group and Boston Python User Group come to mind. I've personally attended a Bay Area R Users Group meetup and it was fantastic. I met tons of interesting people, many of whom were scientists or had studied science in grad school.

Emphasizing followup resources

For the Day 2 wrapup, there could be a brief "after SWC boot camp, where do you go from here?" speech outlining all the different and amazing resources available—office hours, web tutorials, IRC, meetups, etc. Even making people aware of meetup groups and IRC is an accomplishment in itself.

Wrapping up

Volunteering at the boot camp brought my appreciation of Software Carpentry to a new level. I already thought highly of the organization's work after taking the online class two years ago. However, actually being at a boot camp impressed on me the enormous commitment and effort it takes to run a single boot camp—then multiply this by dozens of boot camps that take place in a year.

I very much want to see Software Carpentry succeed. I imagine a future where software skills are part of global science training, and where good software is universally accepted to be a real and important part of scientific results. I will help as much as I can.

(Posted 2013-06-17 by Greg Wilson)

Preston Holmes recently posted a detailed analysis of his experiences helping out at our Salk Institute boot camp. It contains a lot of insights, and is well worth reading.

(Posted 2013-06-15 by Mike Jackson)

Our boot camp at the University of Bristol on September 12-13, 2013, is now open for registration. Please register here if you would like to take part.

(Posted 2013-06-14 by Greg Wilson)

We're very excited to announce the launch of the Mozilla Science Lab, a new initiative that will help researchers around the world use the open web to shape science's future. The Science Lab will foster dialog between the open web community and scientific researchers. Together they'll share ideas, tools, and best practices for using next-generation web solutions to solve real problems in science, and explore ways to make research more agile and collaborative.

The project will be lead by Kaitlin Thaney, a long-time open science advocate. Kaitlin helped found and manage the science program at Creative Commons, and previously worked at Digital Science, a tech company focused on research tools and incubation for science startups. She also advises policymakers on digital infrastructure, data-intensive science, and education to make scientific research more collaborative, open, and reproducible.

For more information, please follow the Science Lab and Kaitlin Thaney on Twitter, or see the announcement from Mark Surman the director of the Mozilla Foundation, or this post by Kaitlin.

We're pretty excited :-)

(Posted 2013-06-13 by Greg Wilson)

Earlier this week, Software Carpentry had its first online lab meeting since October 2012. In attendance were:

We covered a fair bit of ground in 90 minutes; the highlights are below, and comments from both attendees and everyone else would be very welcome (either here or on the discuss mailing list).

Making boot camps more useful:

1. Clearly distinguish boot camps for complete beginners from those for people who already have some coding skills.
2. Do not publicize the boot camp until the curriculum has been specified so that learners can make an informed choice about signing up.
3. Provide a consistent pre-assessment to help people make this decision before they sign up (and to give instructors a clear idea of who they're going to be teaching).
4. Encourage people to teach boot camps spread out over several days or weeks when instructors and learners are co-located.

Sustainability (see this post for background):

1. Poll hosts of previous boot camps to find out what their actual costs were, and whether they would have been willing/able to pay a $1000-$1500 administration fee per boot camp.
2. Depending on the results of that poll, start charging those who can pay an administration fee for setting up boot camps.
3. Explore the PLOS model of different fees and waivers for different countries.
4. Pursue corporate sponsorship (either on a per-bootcamp basis, or as general donations to the cause).
5. Try once again to get funding through existing science training programs.
6. Encourage qualified instructors to teach Software Carpentry commercially, for whatever fees they can charge. (We will not be involved in fee negotiations, and will not "tax" these activities, but would appreciate instructors talking with us before approaching clients so that we're not tripping over each other.)
7. Create a new mailing list for people interested in discussing this. (If you'd like to be added please mail me.)

Consistency:

1. To use our name and logo:
   • At least one qualified instructor.
   • At least two full days (though this may be spread out over a longer period).
   • Cover version control, testing, task automation, and modular program development.
   • Most attendees to date work in science, engineering, medicine, and related fields (we're not teaching web development to salespeople). We are already extending our reach to people in other data-intensive disciplines, but for now at least, the digital humanities are out of reach: we simply don't have enough instructors (or enough experience).
2. To be an instructor:
   • Complete our online training (which includes teaching at least once with a more experienced instructor),
     or
   • Demonstrate extensive relevant teaching experience (and teach at least once with a qualified instructor).
3. These requirements will only affect upcoming boot camps and instructors, not those already scheduled or badged.

Simplifying our shared content, web presence, and workflow.

1. Designate a lead instructor for each boot camp.
2. Streamline the reimbursement process. (Ideally, we want host sites to reimburse instructors directly, but we realize that we'll often be involved, particularly when people are teaching several boot camps in one trip.)
3. Make it much easier for people who are not yet Git experts to update boot camp web pages, contribute teaching materials, etc. We are discussing this on the gits mailing list; please join us there.

Notes:

• Some people proposed having a warm-up day for complete beginners, followed by a regular two-day boot camp that would appeal to people who already have some skills. We'll need to experiment with this to see how well it works.
• There was disagreement over whether testing should remain a required topic: some instructors barely touch on it, and others disagreed over what it ought to mean for day-to-day scientific computing. As Software Carpentry's CDFL¹, I ruled that it is required; however, we need to integrate into the other material rather than teaching it as a standalone topic.

[1] Curmudgeonly Dictator for Life.

(Posted 2013-06-09 by Ben Morris)

Last month, we held two Software Carpentry workshops back-to-back in Durham, North Carolina, one at the National Evolutionary Synthesis Center and a second organized by Cliburn Chan for the biostats group at Duke just three days later. These workshops were taught by Jenny Bryan, Elliott Hauser, and myself with Karen Cranston teaching as well at NESCent. The workshops were both somewhat atypical: NESCent was a smaller workshop (only 11 participants) who work together and share an interest in evolutionary biology, and at both NESCent and Duke we presented in R rather than Python. Overall, both workshops were very successful, and I learned a lot, both about R and teaching a Software Carpentry workshop.

In addition to substituting R for Python, we altered the schedule from what I've seen in previous workshops, teaching R for the entire first day and everything else (shell, git, and Make) on day two. While this meant that Jenny had to present the entire day on day one, her material and instruction style easily held everyone's interest for the whole day. Jenny went at a good pace, moving from basic language essentials and how to interact with RStudio to more complex topics such as data aggregation using the plyr package and producing literate documents using knitr. The R sections, particularly the use of plyr, were consistent crowd favorites. I think the way we rearranged the schedule worked out very well—we avoided confusion by not switching between environments, and using only RStudio on day one allowed us to avoid installation issues. Anyone who had had trouble installing the prerequisites either stayed after on day one or came early on day two for help. We ended up with no setup issues at NESCent and a fair number at Duke (possibly due to inadequate instructions on our boot camp page) but they were easily handled.

We were able, perhaps serendipitously, to present our material in a logically cohesive way that kept students engaged. Instead of every instructor bringing their own material, we used the same files for all sections. The code we wrote during day one was placed under version control on day two, and we wrote a Makefile together to automate our data analysis pipeline and produce an HTML document with code and figures. At NESCent, this wasn't planned ahead of time; I noticed that Jenny had designed her scripts in such a way that they were great candidates to incorporate into my Make tutorial, and so together we wrote one. There was clear interest in Make, so we built on this idea and replicated it at Duke with more preparation.

The number of people attending the NESCent workshop was low, which was in some ways both a pro and a con. On the plus side, we were able to pay more individual attention and Karen led a discussion on data management that was highly relevant to those attending. However, we seemed to run out of steam during the afternoon on day two, where we had planned to address whatever topics participants felt would be helpful but found a lack of enthusiastic suggestions. We corrected this at Duke by being prepared for a more structured presentation. We also found for both the shell and version control sections that our presentation needed to cater to specific tasks that a researcher just learning these skills would use and avoid more complicated topics such as forking and pull requests.

Teaching two workshops back-to-back gave us a chance to refine our approach and materials, and Jenny is looking into how best to incorporate the materials we designed into Software Carpentry for future R workshops to utilize.

NESCent feedback

Duke feedback

(Posted 2013-06-07 by Ted Hart)

Alex Viana and myself recently wrapped up a bootcamp at the National Center for Atmosphere Research (NCAR) in Boulder, CO. It was a lesson in the advantages of being light on your feet when it comes to teaching for Software Carpentry. Our normal audience tends to be scientists with varying degrees of familiarity with the tools that we teach. This time we taught about 20 people all of whom were experience software engineers, some of whom had probably been programming longer than Alex or I had been alive. We had planned on starting the morning with some shell and then moving into Git. A quick survey of the room found that everyone was well versed in the shell so we abandoned that lesson all together and went straight to teaching Git.

Here was where things got interesting. I thought we'd fly through our material because everyone was so experienced. Instead though it almost seemed that we moved more slowly because not only were we teaching Git, but also how git was different than SVN and CVS (other version control systems for the uninitiated). Later in the day we also got into a deep conversation about what types of workflow Git could be used for compared to SVN and CVS.

We had a similar experience the next day teaching Python. Instead of just churning through it we spent a lot of our time talking about how Python was different than other languages the participants programmed in like Java. It turned out that teaching such an advanced crowd meant that we had a group that knew what questions to ask. Topics that we normally gloss over (what is an iterator anyway?) we had to really pick apart because of the knowledge in the room.

As instructors there were two really important lessons for us. The first is never assume that you're the most knowledgeable person in the room. The corollary to that is that just because you aren't doesn't mean you don't have something to teach. We even had a couple of really skilled Python programmers who were happy to be there just to see how they could become better teachers. At first we were a bit intimidated by the depth of knowledge, but really it just forced us to up our game as instructors and I think we're better for it.

The second lesson was that you can never forget that a bootcamp needs to be adaptable. Alex and I were challenged to create curriculum on the spot when we skipped over our 2 hours of the shell. We had to keep in mind that we were there to teach not just stand and deliver a canned lesson even if participants already knew it all. And that's the really great thing about Software Carpentry all around. All the instructors are creative and smart people who are dedicated to being good teachers and don't shy away from these kinds of unexpected challenges. The NCAR bootcamp was definitely my most challenging, but it also offered one of the best opportunities for growth as an instructor.

(Posted 2013-06-02 by Mike Jackson)

On 21st August 2013, we'll be attending INTECOL13 in London's Docklands. The 11th International Congress of Ecology, hosted by the British Ecological Society and INTECOL, is one of the world's premier conference for ecologists. At the invitation of Matthew Smith of the BES Computational Ecology Specialist Interest Group and Microsoft Research, Software Carpentry and The Software Sustainability Institute have a talk and a session.

Steve Crouch will present a talk on behalf of Greg Wilson, on "better computational practices for research". This session will highlight the problems faced by researchers who develop software, and ecologists in particular, including the tractability, development and reproducibility of ecological models, and discuss possible solutions.

After runs at Effective Scientific Programming, Digital Social Research and Dev8D, Mike Jackson will chair another discussion on "what makes good code good". This session will explore what ecologists view as good code, how ecologists learn about good code, the blockers to producing good code and how these can be overcome, by research leaders, institutions, funders, and ourselves.

We look forward to reporting on our experiences.

(Posted 2013-06-07 by Greg Wilson)

It's been eighteen months since we started running two-day boot camps. We've grown rapidly:

Cumulative Number of Boot Camps	Cumulative Enrolment

Now we need to start thinking about how to sustain that growth in the long term. To start, here are some stats on how things have been funded.

First, we've experimented with several different ways to fund travel costs, and several different ways of charging attendees:

Here's how often we've used each combination:

One fact that doesn't show up in this table is that the number of grant-funded boot camps has gone from 16 in 2012 to 2 in 2013. Even if we include two mixed-model boot camps this year in the latter count, we're clearly much closer to boot camps being self-sustaining.

But being closer isn't the same as being there. We're spending about $150K/year in grant money on salaries and overheads to keep this whole show going. At two boot camps a week (which we think we can reach by this time next year), we'd need about $1500 per boot camp to cover those costs. If we stick to an average of 40 learners per boot camp, we'd have to charge a bit less than $40 per person. The problem is, we often can't do this: if we charge for training—even a refundable deposit—then some institutions will charge us for use of their space, and only charging those who don't have such a rule seems unfair. An alternative is to ask for sponsorship on top of instructors' costs, rather than tying it directly to enrolment.

Setting that issue aside for a moment, though, I'm more certain than ever that this training is cost-effective:

1. The average attendee is a graduate student who costs about $50,000/year in salary and overheads (mostly overheads). If she works 40 hours a week, 48 weeks a year, that means her time is worth about $26/hr.
2. Since January 2012, we've run about 75 boot camps for about 2500 people. If each boot camp is two eight-hour days, that works out to $1,040,000 worth of people's time.
3. Instructors' travel costs have averaged about $2000 per boot camp, but we'll round up to $2500 to account for local organizers' time, coffee, etc. That gives us another $187,500.
4. The $150K/year burn rate quoted earlier for salaries and overheads gives us another $225,000, for a total of $1,452,500.
5. But that's only $580 per learner, so at $26/hr, we only have to save people about 22 hours each for this to be worthwhile.
6. Looking at it another way, if instead we save them an hour a week over five years, that's $6240 worth of their time, which is 11 times the up-front investment.

Measured that way, I'd say we're doing more than OK.