Teaching basic lab skills
for research computing

The History of Software Carpentry

Feynman's Final Blackboard

What I cannot create, I do not understand.
– Richard Feynman

In the Beginning...

In 1995-96, Greg Wilson organized a series of articles in IEEE Computational Science & Engineering titled, "What Should Computer Scientists Teach to Physical Scientists and Engineers?" These articles grew out of his frustration working with scientists who wanted to parallelize complex programs but didn't know what version control was, how to write a unit test, or even why they should break their programs down into functions.

In response, John Reynders (then director of the Advanced Computing Laboratory at Los Alamos National Laboratory) invited Wilson and Brent Gorda (now at Intel) to teach a week-long course to LANL staff. The course ran for the first time in July 1998, and was repeated nine times over the next four years. It eventually wound down as the principals moved on to other projects, but taught us two valuable lessons:

  1. There is tremendous pent-up demand for training in basic skills.

  2. Textbook software engineering is not the right thing to teach most scientists.

Going Open

The Software Carpentry materials were updated and released under a Creative Commons license in 2004-05 thanks to support from the Python Software Foundation. They attracted 1000-2000 unique visitors a month, with occasional spikes correlated to courses and mentions in other sites, and were used in a semester-long graduate course offered in 2007-09 at the University of Toronto. Again, we learned some valuable lessons; the most important is that while faculty in science, engineering, and medicine will agree that their students should learn more about computing, they won't agree on what to take out of the current curriculum to make room for it. Until that changes, we have to deliver our lessons "between" standard courses.

The Video Version

Greg Wilson left the University of Toronto in April 2010 to reboot Software Carpentry with support from nine sponsor organizations. Over the next year, he recorded 120 short video lessons and ran half a dozen week-long classes for his backers.

This version of Software Carpentry was much more successful than its predecessors, in part because the scientific landscape itself had changed. Open access publishing, citizen science, and dozens of other innovations had convinced scientists that they needed to be able to do more than just crunch numbers. We also made contact with like-minded organizations, particularly The Hacker Within, who showed us that intensive two-day workshops worked better than week-long classes. Putting this all together led to our current model:

  • Curriculum is developed and improved in a public repository using methods borrowed from the open source software community.

  • Instructors volunteer their time, while workshop host sites cover their travel and accommodation costs.

  • Instructors use live coding instead of slides while learners following along on their own machines.

Scaling Up

In the fall of 2011, Wilson worked with the Mozilla Foundation to prepare a grant to the Sloan Foundation to put Software Carpentry on a more stable financial footing. That grant, awarded in January 2012, and a second one later that year paid for Wilson and some administrative support, which allowed us to increase the number of workshops. In turn, that growth led to us starting a training program to teach instructors the basics of educational psychology and instructional design.

The Present Day

Wilson left Mozilla in July 2014 to help found the Software Carpentry Foundation, an independent non-profit volunteer sponsored by NumFOCUS. Software Carpentry's governing body is a Steering Committee, which is elected from and by its members and assisted by an Advisory Board made up of representatives from partner organizations. The Foundation's first Steering Committee was elected in January 2015, and in October 2015 Jonah Duckles began work as the Foundation's new Executive Director.

In February 2018, Software Carpentry and Data Carpentry merged their projects together into a new project, The Carpentries, sponsored by Community Initiatives. With this merger, Software Carpentry has combined staff, budget and governance to form the new project. The Carpentries continue the work of Software Carpentry and Data Carpentry, realizing that the communities of instructors, members and lesson developers are stronger working together. With over 50 member organizations in 10 countries, The Carpentries seek to build and grow communities of practice around computational skills development for researchers.

Learners

Learners

Workshops

Workshops

Instructors

Instructors

To learn more about our history and the lessons we've learned along the way, please see the paper "Software Carpentry: Lessons Learned".