Empirical Software Engineering

April 24, 2010: We are pleased to announce that Version 4 of this course is now under development. For updates and an early peek at the content, please check out the Software Carpentry blog at http://www.software-carpentry.org/blog/.

1) Is Computer "Science"?

• What kind of science is "computer science"?
  • Theory of algorithms is a branch of mathematics
  • But what about operating systems?
  • Or software engineering?
• Often seems more like:
  • Computer strong opinion
  • Computer well it ought to be that way
  • Computer I heard about this guy once...

2) New Rules

• Growing emphasis since the early 1990s on empirical software engineering [Shaw 1990]
  • Controlled laboratory experiments (for small things)
  • Case studies and data mining (for large ones)
• Much of this lecture comes from [Glass 2002] and [Endres & Rombach 2003]
  • The first may be the most useful little book a software engineer could buy
• Remember: what is true of small systems may not be true of large ones
  • There's a difference between molecules bouncing off each other and fluid turbulence

3) Productivity

• The best programmers are up to 28 times more productive than the worst.
  • Or 5, or 40, or some other number: every quotation seems to use a different value
• Original study was [Sackman et al 1968]
  • Look at the year: most programmers were self-taught, so it wouldn't be surprising if there was wide variation
  • And there were only 12 subjects
  • And "up to" can hide a lot of sins
• [Boehm 1975] claims up to a factor of five
  • Consistent with what's seen in other creative disciplines
• It takes 5000 hours to turn a novice into an expert.
  • Stated in [Norman 1993], but found to be true in many fields
  • FIXME cite the survey paper

4) Complexity

• For every 25% increase in problem complexity, there is a 100% increase in solution complexity.
  • The downside of network effects
  • See [Woodfield 1979] for the original
• Technologists' response is to try to simplify the problem
  • I.e., fit the business to the software
• But some people only have one name
  • And others don't have "first" and "last" names in the Western European sense
  • And every special case in the tax code was vitally important to someone

5) The Mythical Man-Month

• Adding people to a late project makes it later.
  • Because the people already in the project are now spending time getting new staff up to speed
• Originally from [Brooks 1995]
  • First edition appeared in the early 1970s
• If the project is behind:
  • Re-prioritize (triage)
  • Get other people to deal with distractions
    • Counter-productive to have team setting up new servers two weeks before shipping deadline

6) Working Environment

• Environment is a major determinant of productivity.
• Undergraduate labs seem to have been designed to reduce productivity
  • Crowded, noisy, uncomfortable...
  • Infested with IM sessions
• Most cube farms are only a little better
• [Eley & Marmot 1999] summarizes the evidence
• Realistically, there's little you can do

7) Glass's Law

• Any new tool or technique initially makes the adopter slower.
  • It's faster to do today's assignment in Notepad than it is to learn Emacs
• So it's only worth adopting new tools and techniques if you're willing to be patient
  • And have realistic expectations: 5-35% productivity improvement is the best you'll see [Glass 1999]
  • Despite "order of magnitude" claims from vendors or inventors

8) Runaway Projects

• FIXME: the mythology of the "software crisis"
• Data from [Van Genuchten 1991], [Cole 1995], [Hofmann & Lehner 2001] and many other sources

9) Runaway Cause #1: Poor Estimation

• Despite all previous experience, most people plan as if things were going to go perfectly
• Most teams don't have reliable data from previous projects on which to base estimates
• And/or do estimates once, at the beginning of the project, and never update them
  • And estimates often done "on high" and handed down to developers
• Estimates are subject to an anchoring effect [Aranda et al 2007]
  • I.e., estimators are subconsciously biased toward giving customers the answers they want
  • This overshadows experience, technique used, or other factors

10) Runaway Cause #2: Unstable Requirements

• Software seems more malleable than concrete, so it's hard to say "no" to requests for changes
• More up-front analysis usually doesn't solve the problem, because the world won't stand still
  • Software is a victim of its own success
• Agile methodologies are a response
  • Don't plan, adapt
• So is prototyping
  • Catch requirements/design errors before writing lots of code
  • The problem, of course, is convincing management not to ship the prototype...

11) Boehm's Curve

• Worth repeating
• [Boehm & Basili 2001] claim it is 100 times more expensive to fix requirements error in production than in early stages of development
• Missing requirements are the hardest kind of requirement errors to correct.
  • Just as missing code is the hardest to test

12) Reuse

• Software reuse increases productivity and quality.
  • Validated by many experiments and case studies over 30 years
• Small-scale reuse (libraries) is a solved problem
  • Though we're still not very good at version management
• Medium-scale reuse (frameworks) is solved too
  • GUIs, servlets, etc.
• Large-scale reuse (component and system assembly) is still unsolved
  • Web services: someone you've never met, ten time zones away, can break your application without notice
• It takes three times more effort to build a reusable component than it does to build something that's going to be used once
  • Need to use the component in several applications to flesh out its design
  • Have to generalize testing
  • And write real documentation
• If more than 20-25% of a component has to be revised, it's better to rewrite it from scratch
  • Data comes from studies like [Thomas et al 1997]
  • Problem is figuring out ahead of time how much is going to have to be revised...

13) Design vs. Coding

• FIXME: need citations for the multigrid nature of design
• Every design aims for the primitives it's most comfortable/familiar with
  • Problems arise if these aren't the same as those of the person doing the coding
  • Which is why "there's more than one way to do it" is a bad philosophy for a development team
  • And part of why design patterns are so useful
• Good designs require deep application domain knowledge.
  • [Curtis et al 1988]
  • In practice, "requirements elicitation" often means "learning how to think like the customer"
• Strong cohesion and low coupling reduce error rates. [Basili et al 1996]
  • Cohesion: intra-module communication and dependencies
  • Coupling: inter-module communication and dependencies

14) Conway's Law

• A system reflects the organizational structure that built it.
  • Or, if you have four teams, you'll get a four-pass compiler.
• [Herbsleb & Grinter 1999] is just one study
• Reflects the fact that human bandwidth is limited
  • Coupling of code has to reflect people's ability to share information
  • Have to abstract what's at a distance

15) Languages

• Productivity and reliability depend on the length of the program's text, independent of language level.
  • See [Prechelt 2000] for an experiment
  • A direct reflection of human (in)ability to manage detail
• The problem, of course, is that high-level languages run slower
• So build the first version using high-level tools, then profile, then rewrite what you need to

16) Where The Time Goes

• Error removal is the single biggest consumer of time in any software project.
  • Typically 20% on requirements
  • 20% on design
  • 20% on coding
  • 40% on fixing (either immediately or later)
• All that sturdy and agile processes do is change the sizes of the chunks, not their proportions
  • Easy to waste time arguing over whether refactoring counts as "fixing" or not

17) Testing

• Code that developers believe is fully tested typically has only 55-60% path coverage.
  • I.e., less than two thirds of the paths through the code are exercised by tests.
• This is largely because coverage analyzers are still regarded as "advanced" or "exotic"
  • As are test generators that rely on fuzzing or program analysis
• Roughly 35% of errors arise from missing logic paths [Glass 1992]
• Developers test their own code less well than other people.
  • Explaining the problem to a toy hippo can sometimes be enough...
• Suspicion-based testing is more effective than most other approaches.
  • [Hamlet & Taylor 1990] suggests focusing testing on code that:
    • Is written by inexperienced programmers
    • Has a history of failures
    • Failed code inspections
    • Was changed late in the development cycle
    • Developers feel uneasy about

18) Code Inspections

• Rigorous inspections can remove 60-90% of errors before the first test is run.
  • There's that phrase "up to" again...
• But is this more economical than writing tests?
  • Especially since most code is modified several times before being shipped?
• Yes: several independent studies have shown that code inspections are the single most cost-effective error removal technique
  • Pair programming is "continuous code review" (maybe)
  • The "culture of review" in many open source projects is one of the reasons their code is so good
  • But there is no evidence to support the claim that "Given enough eyeballs, all bugs are shallow."
• Effectiveness of inspections is fairly independent of its organizational form.
  • [Porter 1997]
• [Cohen 2006] reports a long study at Cisco

19) There's More Than One Way To Do It

• A combination of different verification and validation methods outperform any single method alone.
• I.e., different techniques catch different (kinds of) bugs [Myers 1978]

20) Maintenance

• Maintenance makes up 40-80% of the cost of a software project.
  • [BOE75] is an early reference, but the finding has been validated many times since
  • The single largest cost in most projects, but almost always underestimated
• Enhancement is roughly 60% of "maintenance".
  • Enhancement usually a result of changing requirements
    • Yes, they keep changing after software is in production
  • How much effort goes into other kinds of maintenance?
    • 18%: adaptive maintenance (i.e., keeping up with a changing environment)
    • 17%: error correction
    • 5%: miscellaneous
• 30% of maintenance time is spent figuring out how the software actually works
  • This figure rises as the software ages
• Better software engineering leads to more maintenance, not less.
  • [Dekleva 1992]: the better the system, the longer it will live, and the more changes are possible
• Small changes have a higher error density than large ones.
  • [Basili et al 1984]
  • Small changes require the same level of program understanding as large ones
    • So mistakes are just as likely
    • So density is higher
  • This does not mean you should batch changes into bigger lumps!
    • But it definitely makes sense to batch the work

21) Bugs

• There are errors that most programmers tend to make.
  • Off-by-one, not handling null, etc.
  • Modern program analysis tools are getting better at catching these
  • No evidence that we're learning not to make them
• Errors tend to cluster.
  • E.g., 80% of defects come from 20% of modules [Boehm & Basili 2001]
    • Which means that if you find a bug, the odds are good that there are other bugs nearby
  • But half of modules are defect-free

22) Summary

• We actually know a lot about software development in the real world
• The problem is, most developers (and managers) don't know what is known
• The next time someone makes a claim, ask them for their evidence
  • And remind them (if necessary) that "evidence" is not the plural of "anecdote"