1) Introduction

• Good programming style is hard to teach because the rules tend to be either banal or overly restrictive
  • "Make methods short, but not too short" isn't useful
  • Neither is, "No method shall be longer than sixty lines."
  • Doesn't help that the strength of programmers' opinions is inversely proportional to the amount of data they have
• But readable code is less likely to contain errors, and much easier to maintain
• This lecture presents and motivates style guidelines
  • For a comprehensive guide to structuring large Unix applications, see [Spinellis 2003]

2) You Can Skip This Lecture If...

• You know what 7+/-2 has to do with readability
• You know what a docstring is, and what it should contain
• You know what traceability is
• You know how to automatically check for style violations

3) Reading is Learning

• People doing creative work in almost every field routinely inspect, dissect, and critique what's come before
  • But most student programmers only ever read short fragments in textbooks
  • Like reading sonnets, then trying to write a novel
• Knowing how to read code is as useful as knowing how to read a proof
  • Have to do it in order to figure out how to make specific changes to specific programs
  • A good way to learn new things

4) Seven Plus or Minus

• The average person's short term memory can hold $7{\pm}2$ items [Hock 2004]
  • Seven random digits (as in phone numbers)
  • Seven tasks that still have to be done
• If we try to remember more than that, we:
  • Make mistakes, or
  • Create chunks so that we can remember things at a higher level
    • Common chord progressions in music
    • "Castled kingside" instead of the positions of five separate pieces
  

  Figure 37.1: Chunking in Short-Term Memory

5) The Mind's Eye

• [Chase & Simon 1973] studied what happened when novice and master chess players were shown actual and random positions
  • Masters remember actual positions better
  • But they were worse at remembering random ones
  • Their minds "see" patterns that aren't there


Figure 37.2: Actual Chess Position



Figure 37.3: Retention of Actual Chess Position



Figure 37.4: Random Chess Position



Figure 37.5: Retention of Random Chess Position

6) What Does This Have to Do With Programming?

• When reading and writing code, you have to keep a bunch of facts straight for a short period of time
  • What do this function's parameters mean?
  • What does this loop's index refer to?
• The more odds and ends readers have to keep track of, the more errors they will make
  • Goal of style rules is therefore to reduce the number of things the reader has to juggle mentally
• The greater a difference is, the more likely we are to notice it
  • So every semantic difference ought to be visually distinct...
  • ...and every difference in naming or layout ought to mean something
• Most important thing is to be consistent
  • Anything consistent is readable after a while
  • Just watch kids learning to read French, Punjabi, and Korean

7) Python Style Guide

• Taken from [Python Style Guide]
  • Stick to this unless you have hard data that proves something else is better
• Basic layout
  • Indent blocks using four spaces
  • Keep lines less than 80 characters long
  • Separate functions with two blank lines
  • Separate logical chunks of long functions with a single blank line
  • Put comments on lines of their own, rather than to the right of code
• Rules relating to classes will become clear Python Basic Object-Oriented Programming

8) Naming

• Names of files, classes, methods, variables, and other things are the most visible clue to purpose
  • A variable called temperature shouldn't be used to store the number of pottery shards found at a dig site
• Choose names that are both meaningful and readable
  • current_surface_temperature_of_probe is meaningful, but not readable
  • cstp is easier to read, but hard to understand...
  • ...and easy to confuse with ctsp
• If you must abbreviate, be consistent
  • curr_ave_temp instead of current_average_temperature is OK...
  • ...but only if no one else is using curnt_av_tmp

9) Scope and Size

• The smaller the scope of a name, the more compact it can be
  • It's OK to use i and j for indices in tightly-nested for loops
  • But not OK if the loop bodies are several pages long
    • Of course, they shouldn't be anyway...
• The wider the scope of a name, the more descriptive it has to be
  • Call a class ExperimentalRecord, rather than ER or ExpRec

10) The Difference It Makes

• Before:

import sys, os
import reader, splitter, transpose
a=[]
b=[]
c=[]
d=sys.argv[1]
a=reader.rdlines(d)
b=splitter.splitsec(a)
c=d.split('.')
for i in range(len(b)):
    if os.path.isfile('%s.%d.dat'%(c[0],i+1)):
        print '%s.%d.dat already exists!'%(c[0],i+1)
        break
    else:
        output=file('%s.%d.dat'%(c[0],i+1),'w')
        print>>output,transpose.txpose(b[i])
        output.close()

• After:

import sys, os
import reader, splitter, transpose

input_file_name = sys.argv[1]
lines = reader.read_lines_from_file(input_file_name)
sections = splitter.split_into_sections(lines)
file_name_stem = input_file_name.split('.')[0]
for i in range(len(sections)):
    output_file_name = '%s.%d.dat' % (file_name_stem, i+1)
    if os.path.isfile(output_file_name):
        print '%s already exists!' % output_file_name
        break
    else:
        output = file(output_file_name, 'w')
        print >> output, transpose.transpose(sections[i])
        output.close()

11) Function Length

• Every function should do exactly one job
  • Should be able to describe that job in a single memorable sentence
  • If that sentence is five phrases joined with "and", the function should be split up
• A good way to judge size and scope is the notion of a program slice
  • The subset of names in scope at a particular statement that are needed to understand what that statement does
  • If the slice is much smaller than the method itself, the method is probably bloated
• A thousand one-line methods are not an improvement over one thousand-line method

12) What Does This Function Do?

• Key questions:
  • What are the function's arguments for?
  • What does it return?
  • What side effects does it have?

# What's missing, and what's extra?
def diff_filelist(dir_path, manifest,
                  ignore=[os.curdir, os.pardir, '.svn']):

    def show_diff(title, diff):
        if diff:
            print title
            for d in diff:
                print '\t' + d

    expected = Set()
    inf = open(manifest, 'r')
    for line in inf:
        expected.add(line.strip())
    inf.close()

    actual = Set()
    contents = os.listdir(dir_path)
    for c in contents:
        if c not in ignore:
            actual.add(c)

    show_diff('missing:', expected - actual)
    show_diff('surplus:', actual - expected)

13) Ways to Answer the Question

• Read comments
  • Not particularly helpful in this case
• Guess based on names
  • Find the difference between two file lists?
  • But dir_path suggests "directory path"
• Read the function body
  • Define a helper function that displays diff if it isn't empty
  • Make a set holding the lines in the manifest file
  • Make a set holding the contents of the directory specified by dir_path
    • Ignoring certain things
  • Use the helper function to show the differences between these two sets
• Notice how "explaining" is the same as "creating chunks"

14) Other Sources of Information

• External documentation
  • There is none
• Calls to the function
  • One in the bottom of the same file

if __name__ == '__main__':
    if len(sys.argv) != 3:
        print >> sys.stderr, "usage: diff_filelist directory_path manifest_file"
        sys.exit(1)
    diff_filelist(sys.argv[1], sys.argv[2])

• Use grep or "Find in Files" to search for others
  • A few examples of how something is used is often sufficient
  • Except when it's misleading

15) Idioms

• Every language (human or computer) has idioms
  • E.g., "face the music" does not mean "look at the orchestra"
• Older versions of Python couldn't iterate directly over a file using for line in input:
  • Solution: an "infinite" loop with a break to handle end-of-input

count = 0
while 1:
    line = infile.readline()
    if not line:
        break
    count += 1

• Uses 1 instead of True because older Pythons didn't define True

16) Learning Idioms

• You can figure out what the code is doing even if you don't know the idiom...
  • ...but knowing the idiom makes it a lot simpler for your brain to chunk what it's reading
• Learn idioms by reading:
  • Books (e.g., the "Effective" series from Addison-Wesley)
  • Other people's code

17) Style Tools

• Many tools exist to check and enforce style guidelines
  • Indentation and naming
  • Function and module length
  • Dead and duplicated code
  • Failure to check for errors
  • Make these tools part of your project's build
    • Don't check anything into version control unless it passes a style check
    • Take their warnings very seriously when debugging
    • If someone is sloppy enough to make style mistakes, they're probably making serious mistakes too

18) Python Style Tools

• [PyLint] parses programs to create an abstract syntax tree


Figure 37.6: Abstract Syntax Tree

• Then searches the tree for matches to problem patterns
• [PyChecker] imports the module (or modules)
  • If the code doesn't run, [PyChecker] can't analyze it
• Both tools allow users to:
  • Customize built-in checks (e.g., specify maximum length of a function)
  • Define entirely new checks

19) Documentation

• Requirements
  • What needs the software is supposed to meet
  • "If more than 100 events arrive in a second, the seismograph interface must store them in a queue" tells you to look for a seismograph interface, and a queue that feeds it data
• User guide
  • Actually just another way to specify requirements
• Architectural descriptions
  • Architecture is what you draw on the whiteboard when explaining the program to other people
  • Shows relationships between major modules, data flow, etc.
  • Gives other programmers a mental map of how everything fits together

20) More On Documentation

• At start of functions (or classes, or methods) to explain what they're for
  • "This function returns the first pair of non-overlapping subsequences that match the input pattern, or null otherwise"
  • The programmer's guide to the software
• Embedded in code to explain tricky bits
  • "This is a while loop, instead of a for, because it may delete items from the list as it goes"
  • In general, if you need to explain the code, you ought to simplify it instead

21) Traceability

• Traceability is the key to reproducibility
  • ...which is the key to debuggability
  • Where did your data come from?
  • What did you do to it?
  • Using which versions of which programs?
• Always include version control information in source files
  • Most version control systems will update text like $Revision: 421$ when you submit changes
  • The file carries its identity with it when it's printed, archived, or emailed
• Usually put the keyword inside a string, so that version information is available inside the program
  • By convention, Python calls this string __version__

__version__ = "$Revision: 423$"

if __name__ == '__main__':
    print __version__

22) Tracing Data

• Every original data file should go under version control
  • Record author and date of last change as well as revision number
• Carry this information through when processing files
  • E.g., input file biomes.dat has a header $Revision: 421$
  • Is processed by version 37 of ecoanalyzer.py
  • So include the following comment header in the output file

# $Revision: $
# From: biomes.dat version 421
# By: ecoanalyzer.py version 37
# Parameters: sliding_average 20 trim False
# On: 2006-02-22 12:14:07 EST

• The $Revision:$ header will be expanded when the file is first checked in

23) Embedding Documentation

• Embedded documentation is more likely to be up to date than external documentation
  • A simple form of literate programming
  • It's right in front of programmers as they make changes
• Javadoc translates specially-formatted comments into HTML

/**
 * Returns the least common ancestor of two species based on DNA
 * comparison, with certainty no less than the specified threshold.
 * Note that getConcestor(X, X, t) returns X for any threshold.
 *
 * @param left        one of the base species for the search
 * @param right       the other base species for the search
 * @param threshold   the degree of certainty required
 * @return            the common ancestor, or null if none is found
 * @see               Species
 */
public Species getConcestor(Species left, Species right, float threshold) {
    ...implementation...
}

<p><strong>getConcestor</strong></p>

<p><code>public&nbsp;Species&nbsp;getConcestor(Species&nbsp;left,&nbsp;Species&nbsp;right,&nbsp;float&nbsp;threshold)</code></p>

<blockquote>

<p>Returns the least common ancestor of two species based on DNA
comparison, with certainty no less than the specified threshold.  Note
that getConcestor(X, X, t) returns X for any threshold.</p>

<p><strong>Parameters:</strong></p>
<blockquote>
<p><code>left</code> - one of the base species for the search</p>
<p><code>right</code> - the other base species for the search</p>
<p><code>threshold</code> - the degree of certainty required</p>
</blockquote>

<p><strong>Parameters:</strong></p>
<blockquote>
<p>the common ancestor, or null if none is found</p>
</blockquote>

<p><strong>See Also:</strong></p>
<blockquote>
<p><code>Image</code></p>
</blockquote>

</blockquote>

24) Docstrings

• Python uses documentation strings (or docstrings) instead of comments
  • A string at the start of a module or function that isn't assigned to anything becomes the object's __doc__ attribute
  • Unlike a comment, it's there at runtime

'''This module provides functions that search and compare genomes.
All functions assume that their input arguments are in valid CCSN-2
format; unless specifically noted, they do not modify their arguments,
print, or have other side effects.
'''

__version__ = '$Revision: 497$'

def get_concestor(left, right, threshold):
    '''Find the least common ancestor of two species.

    This function searches for a least common ancestor based on DNA
    comparison with certainty no less than the specified threshold.
    If one can be found, it is returned; otherwise, the function
    returns None.  get_concestor(X, X, t) returns X for any threshold.

    left      : one of the base species for the search
    right     : the other base species for the search
    threshold : the degree of certainty required
    '''

    pass # implementation would go here

>>> import genome
>>> print genome.__doc__
This module provides functions that search and compare genomes.
All functions assume that their input arguments are in valid CCSN-2
format; unless specifically noted, they do not modify their arguments,
print, or have other side effects.
 
>>> print genome.get_concestor.__doc__
Find the least common ancestor of two species.

    This function searches for a least common ancestor based on DNA
    comparison with certainty no less than the specified threshold.
    If one can be found, it is returned; otherwise, the function
    returns None.  get_concestor(X, X, t) returns X for any threshold.

    left      : one of the base species for the search
    right     : the other base species for the search
    threshold : the degree of certainty required

• Python's [Docutils] will extract, format, and cross-reference docstrings

25) Summary

• Code and documentation decay over time [Eick et al 2001]
• To prevent this, must:
  • Make good style a habit
  • Back it up with automated checks
• Remember, we don't actually write programs for the benefit of computers
  • It takes a lot of very sophisticated software to translate our programs into a form computers understand
• We write programs for other people
  • Our colleagues
  • Our future selves