Fork me on GitHub

Sloan Foundation Proposal Round 2

Posted 2013-07-05 by Greg Wilson in Sloan Foundation.

From our "better late than never" department: we submitted a proposal to the Sloan Foundation last August to create what we're now calling the Mozilla Science Lab. As regular readers already know, it was accepted; we're now more than six months into the project, so in keeping with our own earlier practice, and with the examples of ImpactStory, rOpenSci, NISO, and Dr. Holly Blik, I'm posting the proposal here. I hope it's useful to other people seeking funding for ventures like ours, and I'd be happy to answer questions.


1.) What is the main issue, problem, or subject and why is it important?

Created by Scientists: The web was created by a scientist to accelerate science. But nearly 30 years on, only a small minority uses the web for more than publishing and distribution. The web has fundamentally altered the nature of commerce, education, culture, and civil society through open communication, access to information, and generative innovation at a global scale. But with a few exceptions, most scientific computing is still done by individual scientists crunching numbers offline.

More Effective, but Not Transformed: Software Carpentry set out to help scientists use computing and the web more effectively in their research. We developed curriculum and ran workshops that taught basic skills such as version control, testing, and program design. Our approach has proven effective: participants become more productive, confident in their ability to tackle new challenges, and gain awareness of what is possible. However, they still return to their labs to write their own code, for their own data, by themselves. The web remains a platform to discuss and disseminate results, but not a tool to conduct, strengthen, and expand their work.

The Methodology of the Web: The web is a technology stack: mark-up languages, hyperlinks, and network protocols. But the web is also a set of cultural norms and practices. Its architecture makes it an open lab where anyone can 'crack the hood' and see how something was done. This has bred a culture that sees individual accomplishment as result to be celebrated, but also a model to be emulated, material to be repurposed, and a contribution to a shared endeavor. In this way, webmaking and scientific practice have much in common. So why have scientists not embraced the web?

Three New Blockers: The first issue is skill. In this, Software Carpentry is correct and proving effective. But there are three other challenges:

Technology: Tools for doing science on the web do exist, but most require a sophistication that puts them out of the reach of most working scientists.

Practice: The value of working in the open — APIs, sharing reusable code, mass collaboration, distributed computing — can seem in conflict with established norms around priority, publishing, and attribution.

Scale: A shortage of working models, knowledgeable instructors, and on-ramps to engagement inhibit growth. What exemplars exist have not coalesced to attract converts.

Science that Works Like the Web: We believe knowing how to code — how to create content and work on the web — is a core literacy akin to reading or mathematics. But more than that, we view the web as a 21st century tool, mindset, and force transforming the nature of professional practice. Working not just on, but like the web can accelerate scientific research: data sets become live streams rather than static files; small ideas grow to become collaborative frameworks; and researchers gain access to thousands of participants willing to report bird sightings, analyze old ships' logs, or classify galaxies. Science pioneered open collaboration and the gift economy, but the web has perfected it. We will build from Software Carpentry to re-establish this generative interplay.

2.) What is the major related work in this field?

Two fields of practice relate to our approach: delivering computing and webmaking skills to non-programmers (including scientists), and conducting science on the web.

Teaching Scientists to Code: Several studies (see bibliography) of how scientists use computers and the web, including the largest by Dr. Wilson in 2008-09, have found that most scientists learn what they know about these subjects through osmosis and word of mouth. Most training meant to address this issue does not target scientists' specific needs, places too much emphasis on programming and number crunching, and/or jumps to advanced topics before scientists have mastered the basics. Software Carpentry is the most significant effort to date to address the issue. Since its inception in 1997, the program has taught several thousand researchers the concepts, skills, and tools they need to get more done in less time and with less pain.

Two other projects of note are the Next-Generation Sequencing (NGS) summer school run by Prof. C. Titus Brown at Michigan State University, and the training and seminars offered by the Software Sustainability Institute (SSI) at the University of Edinburgh. Brown, who has also taught Software Carpentry, starts with the same material and continues into bioinformatics-specific topics. The SSI, which was established to train scientists how to write and collaborate around reusable software, has been working closely with Software Carpentry since January 2012.

Beyond science, Mozilla runs complementary 'open labs' that work to build the skills and transform the practice of professions such as journalism (Open|News) and filmmaking (Living Docs). Two years in, these programs are meeting with significant success. The community behind the Living Docs lab has produced Popcorn, a technology that lets video files control web content and vice versa. Wired and Fast Company both called Popcorn the future of online video. Living Docs has attracted partnership and investment from the Tribeca Film Institute, ITVS, and the Ford Foundation. Popcorn is also driving change within industry. Fox Sports and ABC have commissioned Popcorn-based content, and a start-up just completed a $2M venture capital round to launch a business anchored on the technology. Countless hackjams bring together filmmakers, designers, and developers at almost every major film festival. Open|News has achieved similar results, with partners such as the New York Times, BBC, Al Jazeera, and ProPublica working on the web to produce new tools, including HyperAudio Pad that lets audio files be remixed by editing written transcripts, and DocumentCloud, which embeds source materials into digital news stories (more projects at mozillaopennews.org/code.html). Finally, the labs benefit from and work with each other, as evidenced by PBS NewsHour's plans to use Popcorn for their digital coverage of the 2012 elections.

Conducting Science on the Web: Mozilla's efforts are part of an emerging movement to bring science to the web. From pan-science approaches such as the Panton Principles and calls to publish scientific data as open data, to field-specific initiatives such as Blue Obelisk in chemistry, scientists are taking tentative steps towards working on the web. More progress has been made on projects that benefit from engagement with citizen scientists. In addition to the renowned SETI@Home project, Galaxy Zoo has led to the discovery of new classes of galaxies and yielded a wealth of data. The web-based platform built for the project has expanded to support everything from studying climate change using data from ships' logs to categorizing the speech patterns of killer whales. Another related venture is the emerging discipline of data science: ways of manipulating, mixing, and visualizing the enormous volumes of open data. Projects such as D3 and Wrangler (both built by Prof. Jeff Heer's group at Stanford) are using web-native technologies to make high-quality, hackable data visualizations. These tools can be used to conduct advanced research, but also provide accessible and easily understood on-ramps to help people realize the value of open data, visualizations, and web-enabled science.

3.) Why is the proposer qualified to address the issue or subject for which funds are being sought?

Mozilla Foundation: Mozilla is a non-profit organization best known as the makers of Firefox. At the core of our work are programs that move people from using the web to making the web: from consumption to creation. This involves building tools and software, but also helping people learn about the open ethos and building blocks that make up the web. We run a number of programs related to the work detailed in this proposal:

Software Carpentry: Our ongoing project with Sloan to build the technological skill set of scientists and graduate students, detailed in section 8.

Webmaker: The umbrella program for our work related to learning (including Software Carpentry), Webmaker is a set of tools, curriculum and assessments, and communities of practice dedicated to teaching millions of people how to code, shape, and remix content for the web. Webmaker's design is underpinned by significant research into the skills taxonomy behind digital literacy; new and peer-based models of instruction, assessment, and accreditation (Open Badges); and a growing pool of partners including the New York Times, NASA, Department of Education, the National Science Foundation, and the Tribeca Film Institute.

Open|News, Living Docs & Hive NYC: Parallel programs that, like the work detailed in this proposal, build content, tools, and partnerships for and with participants from specific communities — in these instances journalists, filmmakers, and youth.

We also run the Mozilla Developer Network, the largest repository of resources on how to build content for the web; the Mozilla Reps program, comprised of hundreds of volunteers who host events around the world; Mozilla Ignite, a project with the White House and National Science Foundation to drive the development of gigabit-per-second networks; and we continue to engage in top-level strategy conversations that shape the future of the web. Each of these resources will support our emerging work with the scientific community.

Dr. Greg Wilson: Greg will lead the work on Software Carpentry. A 25-year veteran of the software industry, Greg was co-winner of the Jolt Award for Best General Technical Book in 2008 and received ComputerWorld Canada's 'IT Educator of the Year' award in 2010. He started Software Carpentry as a training course at Los Alamos National Laboratory in 1998. Greg holds a PhD in Computer Science from the University of Edinburgh.

Erin Knight: Erin, Mozilla's Senior Director Learning, leads Mozilla's work in the field, which includes building the content and community needed to create a generation of webmakers. Previously, Erin served as the Research Director at the Center for Next Generation Teaching and Learning. Erin holds an MA in Information Systems and Management from the University of California, Berkeley.

Dr. David Ascher: David's role at Mozilla focuses on the exploration and development of new product initiatives within the Mozilla Labs group. David's previous experiences include academic research in cognitive science, as well as leading the development of Thunderbird.

4.) What is the approach being taken?

This project will see us engage in a three-phase approach.

1. Expanding Software Carpentry to teach more scientists how to work on the web;

2. Building an open Webmaking Science Lab, led by a vibrant and respected scientist, to drive the development of tools, practices, and community around science on the web; and

3. Challenging the lab to engage scientists, developers, and designers to build tools and learning content (products) that address research needs and provide paths to broader participation.

Phases I and II are covered by this proposal, with prototyping funding for Phase III. Should Phase I and II launch successfully, we would work with Sloan to engage additional private and public funders, including the NSF, JISC, and Moore.

1. Deliver More Training to More Scientists: Software Carpentry is working. Independent assessment of the pilot phase found that scientists enjoyed the training and that it improved their skills (see question 8). Workshops are almost always over-subscribed and alumni are beginning to run follow-on tutorials on their own. Given this success, we want to increase our ability to reach more participants, expand the training to teach scientists how to use the web in their research, and provide for longer, ongoing support. In this way, SWC will improve the research practices of individual scientists and build the constituency of individuals able to contribute to the advancement of science on the web.

Our Approach:

How We'll Do It:

2. Launch a Webmaking Science Lab: The subject area we are planning to tackle — science and the web — is huge and complex. Our understanding of how science could use the web, what learning is useful, and the tools needed to support their work is going to evolve quickly. What we need is a process — an engine — to drive continued innovation, produce tools, explore new practices, and grow the number of scientists, developers, and designers collaborating over the long term.

Our Approach:

How We'll Do It:

3. Design and Ship Products for Science on the Web: Open Labs are community-driven product development environments. Participants collaborate to produce software, tools, learning materials, and flagship projects demonstrating their application. This product development ecosystem is driven by motivated participants from diverse backgrounds, in this instance scientists, developers, designers, hobbyists, students, and the public. Within the scope of this proposal, we will undertake to produce small prototypes and reference implementations to test potential directions and underpin future proposals. Should we be successful at building a community of scientists, developers, and designers around a meaningful product direction, we will work with Sloan to approach other funders to bring resources to the development activities taking place within the Lab.

5.) What will be the output from the project?

Software Carpentry

Deliverables: An expanded curriculum that incorporates webmaking for scientists; training materials and specialized 'train the trainers' workshops for instructors; and a community to support them.

Objectives & Metrics

Webmaking Science Lab

Deliverables: A community of practice, including Software Carpentry alumni; integration with Mozilla's broader community-driven innovation events, including hackjams and the Mozilla Festival; and ideas, tools, practices, and lessons that emerge from Lab experiments.

Objectives & Metrics

6.) What is the justification for the amount of money being requested?

Program Direction: Mozilla's Sr. Director of Learning, Erin Knight, and Sr. Director of Software, Dr. David Ascher, will serve as the lead executives overseeing the project. Our Director of Communications, Matt Thompson, and Manager of Global Events, Michelle Thorne, will coordinate and lead outreach and communications, as well as integration with Mozilla's broader events and community-driven innovation activities. We will provide for these costs from our existing revenue. Within the requested amount, an external consultant will be engaged to lead assessment, evaluation, and reporting efforts.

Project Delivery: The work on Software Carpentry will continue to be led by Dr. Wilson. He will oversee and co-author upgrades to the curriculum, develop the webmaking workshop, recruit new workshop hosts and instructors, and build the infrastructure (materials, training sessions, peer support networks, etc.) needed to scale Software Carpentry to thousands of participants. Dr. Wilson will also leverage his experience in academia to foster institutional recognition for the importance of proper technology practices in the conduct of scientific research. Greg will be supported by a dedicated Workshop Facilitator. The Facilitator will deliver some workshops, but will also match instructors to venues, deliver online tutorials, facilitate peer-to-peer mentoring, and manage credentialization. Dr. Wilson and the Workshop Facilitator will co-author the guides and other materials required to expand the scope and participation level in Software Carpentry.

For the Webmaking Science Lab, we will engage the leadership of a distinguished and high-profile scientist to lead the design and development of the Lab and its activities. The individual will reach out to the scientific community and engage them in shaping the purpose, scope of activities, and product vision for the Lab. There will also be a small pool of funding to develop prototypes and reference implementations of potential products. We will also work to secure the resources required to augment the Lab with additional development support.

Wage Costs, Benefits & Overhead: The 30% multiplier includes 7.5% overhead and 22.5% wage costs and benefits.

Travel: We have allowed for one trip per month for Software Carpentry staff, and one trip per quarter for the Webmaking Science Lab Director.

7.) What other sources of support does the proposer have in hand or has applied for?

Mozilla Webmaker: Our work with Sloan will fall within our larger Webmaker program. Funding for Webmaker comes from current and prior revenues received as the shareholder of the Mozilla Corporation; funding from organizations including the Ford Foundation, National Science Foundation, and MacArthur Foundation; and Join Mozilla, a broad-based fundraising and community-organizing effort.

Software Carpentry: Since the start of our work on Software Carpentry this January, we have received money to support travel costs for workshop instructors from Enthought Inc. (a geophysics and computational financial consultancy based in Austin and New York City); from the Sound Software research consortium based at Queen Mary, University of London, to support development of an instructor's guide and delivery of a workshop; and Prof. Brown at Michigan State University has received funding for Software Carpentry as part of a larger NSF proposal. Going forward, we have verbal confirmation of support from Lawrence Berkeley National Laboratory to provide both basic and advanced (web-oriented) workshops, and Prof. Carole Goble of the University of Manchester is including support for Software Carpentry in the UK's proposal for the European Union's ELIXIR program.

8.) What is the status and output of current and/or previous Sloan grants?

In January 2012, the Sloan Foundation provided a grant of $124,625 in support of Software Carpentry. Six months later, we have run 20 workshops for over 600 scientists and two independent assessments have shown significant positive impact. We have made substantial progress on institutional engagement and have shown that our approach is scalable: a quarter of workshops have been peer-led, and we now have over a dozen instructors able to deliver those workshops and online tutorials.

What We Planned: Our original proposal for Software Carpentry focused on four metrics:

  1. The percentage of course participants that offer or plan to offer in-person workshops and study groups on their own, as well as recommend the course to their peers. 96% of the participants surveyed said that they would recommend the workshop to others and 16 student-led groups have committed to hosting workshops in the coming months.
  2. The scope of badge display and associated reputation. As detailed below, we shifted our focus from credentialing learners to credentialing instructors. Most are further along in their careers and find recognition of special accomplishments valuable as they apply for post-doctoral or faculty positions. We expect to be able to report on this shift in the first half of 2013.
  3. The number of institutions engaging in our work and the depth of that engagement. Faculty members at Michigan State University and Utah State University are asking their universities to count Software Carpentry as fulfillment of the National Science Foundation's requirements for training in Responsible Conduct of Research (RCR). If this is approved, we expect a significant increase in interest at those institutions, nearly 100% uptake of badges, and emulation at other institutions. In addition, Software Carpentry is working with the Software Sustainability Institute in the UK to formulate a 'driver's exam' for the new DiRAC high-performance computing facility. The aim is to provide formative assessment for scientists who wish to use DiRAC via an hour-long examination that will include written, recorded, and interactive components.
  4. The number of and degree to which former students becoming makers. We initially planned to have learners create new examples and lessons for inclusion in our web-based material. What we found was that people who are interested in contributing would much rather do so in person, e.g., by helping out with workshops or online tutorials. As a result, we shifted our focus to recruiting and training the next wave of instructors. 36 people have helped deliver Software Carpentry workshops, half of them independently, and five of those have run online tutorials reaching dozens of people each.

Where We Shifted: We had planned to develop 15 hours or more of new video content during the pilot. However, it became clear by April 2012 that this would not add much value in the eyes of our intended audience. We therefore re-prioritized as follows:

These adjustments were decided and managed in accordance with Mozilla's agile and iterative approach to program implementation.

How We Did: We evaluated the effectiveness of the Software Carpentry program through several means. The bulk of the work was performed by Dr. Jorge Aranda, who surveyed and interviewed participants, observed a workshop, and analyzed screencasts of participants working through a programming assignment. Dr. Aranda's survey found significant increases in participants' understanding and use of shell commands, version control tools, Python, and testing techniques. Perhaps more importantly, participants reported better proficiency with software tools; greater concern for issues of provenance and code quality; better strategies to approach software development; and new research questions that have become accessible thanks to an increase in participants' software development skills.

Independently, Prof. Julie Libarkin (Michigan State University) performed a more detailed assessment of participants in a workshop held there, which was attended remotely by students from the University of Texas at Austin. 85% of participants reported that they learned what they hoped to learn, 81% changed their computational understanding, and 96% said they would recommend the workshop to others.

References

  1. Jeffrey C. Carver, Richard P. Kendall, Susan E. Squires, and Douglass E. Post: "Software Development Environments for Scientific and Engineering Software: A Series of Case Studies." 29th International Conference on Software Engineering, 2007.
  2. Jo Erskine Hannay, Hans Petter Langtangen, Carolyn MacLeod, Dietmar Pfahl, Janice Singer, and Greg Wilson: "How Do Scientists Develop and Use Scientific Software?" Second International Workshop on Software Engineering for Computational Science and Engineering, 2009.
  3. Prakash Prabhu, Thomas B. Jablin, Arun Raman, Yun Zhang, Jialu Huang, Hanjun Kim, Nick P. Johnson, Feng Liu, Soumyadeep Ghosh, Stephen Beard, Taewook Oh, Matthew Zoufaly, David Walker, and David I. August: "A Survey of the Practice of Computational Science." 24th ACM/IEEE Conference on High Performance Computing, Networking, Storage and Analysis, 2011.
  4. James Howison and James D. Herbsleb: "Scientific Software: Production and Collaboration." Computer Support for Cooperative Work, 2011.
  5. Judith Segal: "When Software Engineers Met Research Scientists: A Case Study." Empirical Software Engineering, 10(4), 2005.
comments powered by Disqus