Functions and Libraries in Python

def zero():
    return 0

result = zero()
print "zero produces", result
zero produces 0

followed by the function’s name. The empty parentheses signal that the function doesn’t take any inputs; we’ll come back to that in a moment.

The body of the function is then indented, just like the body of a loop. We use the keyword return to signal the end of the function; the value after return is what the function produces.

We have seen lots of examples already of calling functions,  so the third line of code look familiar: when Python sees the statement result = zero() it sets aside whatever it was doing, goes and does whatever zero tells it to do, and then substitutes the value that zero returns for the call. In this case, that means assigning the value to the variable result, which is then printed.

Functions that always produce the same value aren’t particularly interesting (though they can actually be useful, in ways which we’ll see later). Most functions accept input values, or parameters, and use those in whatever calcluations they do, so that their final result depends on what they were given to work with.
For example, this function turns a temperature in Fahrenheit  into a temperature in Kelvin:

def fahr_to_celsius(temp):
    k = fahr_to_kelvin(temp)
    return kelvin_to_celsius(k)

temp_f = 32.0
temp_c = fahr_to_celsius(temp_f)
print 'water freezes at', temp_c
water freezes at 0.0

This is exactly like what we do all the time in mathematics: if we know how to find the logarithm of a number, and how to calculate its sine, we can combine the two and define a new function that calculates sin(log(x)).

Combining fahr_to_kelvin and kelvin_to_celsius is no different. In order to really understand what happens when we do this, we need to understand the function call stack, or just “stack” for short.

Here are the three function definitions once again:

It should now be clear why we got the error we did, and why we got it where we did. When Python is on line 6, the uppermost frame of the stack doesn’t contain a variable called temp: it was discarded when the call to kelvin_to_celsius finished executing.

To understand why Python (and other languages) go through all of this, let’s go back to fahr_to_celsius again. Its definition, and the definitions of the functions it calls, are:

via the following steps:

If sum‘s result and square‘s result were the same variable, though, we would get 8 instead:

If we change the name of the variable in square from result to y, the final answer will be 5 again. Changing the name of a variable shouldn’t matter: f(x)=x² and f(y)=y² ought to calculate the same value, and if changing a variable name in one part of our program can change the result calculated by another, we will have to keep the entire program in our head in order to make any change safely.

The fundamental issue here is one of evolution rather than one of technology. Human short-term memory can only hold a few items at a time; the value is sometimes given as “seven plus or minus two”, and while that is an over-simplification, it’s a good guideline. If we need to remember more unrelated bits of information than that for more than a few seconds, they become jumbled and we start making mistakes. This means that if we have to keep a dozen things straight in our mind in order to understand or change a piece of code, we will make mistakes—a lot of them. Most programming languages therefore enforce a “local scope only” rule so that programmers can ignore what’s inside the functions they are calling and use that part of short term memory to keep track of the function they’re writing (or reading) instead.

There is one important pragmatic exception to the “local scope only” rule. Every function also has access to the global scope, which is all the top-level definitions in the program (i.e., ones that aren’t inside any particular function).

In our pictures, the global scope is the bottom-most frame on the stack, the one that is there when the program starts and never goes away. Functions need access to the global scope because that is where other functions are defined. Going back to our temperature example, all three of our functions live in the global scope, if fahr_to_celsius could only see names defined in its local scope, it wouldn’t be able to see either fahr_to_kelvin or kelvin_to_celsius, and therefore wouldn’t be able to call them.

Programmers often put constants at the top level of their program so that they don’t need to pass them into functions.
For example, it’s common to see code like this:

How could you fix this?

Multiple Arguments

def average3(a, b, c):
    return (a + b + c) / 3.0

x = 1
y = 2
z = 5
a = average3(x, y, z)
print a
2.66666666667

print average3(10, 20, 50)
26.6666666667

print average3(1, 5)
Traceback (most recent call last): File "src/funclib/average-3-wrong.py", line 4, in <module> print 1, 5, '=>', average3(1, 5) TypeError: average3() takes exactly 3 arguments (2 given)

def average3(a=0.0, b=0.0, c=0.0):
    return (a + b + c) / 3.0

print '()', average3()
print '(1.0)', average3(1.0)
print '(1.0, 2.0)', average3(1.0, 2.0)
print '(1.0, 2.0, 5.0)', average3(1.0, 2.0, 5.0)
() 0.0 (1.0) 0.333333333333 (1.0, 2.0) 1.0 (1.0, 2.0, 5.0) 2.66666666667

def average_list(values):
    result = 0.0
    for v in values:
        result += v
    return result / len(values)

for test in ([1.0], [1.0, 2.0], [1.0, 2.0, 5.0]):
    print test, '=>', average_list(test)
[1.0] => 1.0 [1.0, 2.0] => 1.5 [1.0, 2.0, 5.0] => 2.66666666667

a = average_list(values[20:90])

but another would be to allow them to tell average_list what range to use. This is what most list and string methods do: if they are passed one value, they work from that index to the end of the data, while if they are passed two, they work on the range those indices delimit.

Try this!

If the language we are using doesn’t let us define default parameter values, we could turn our function into three:

def average_list(values, start=0, end=None):
    if end is None:
        end = len(values)
    result = 0.0
    i = start
    while i < end:
        result += values[i]
        i += 1
    return result / (end - start)

The initial if statement’s job is to spot this case and re-set end to the length of values so that the loop that does the averaging will run correctly. Finally, if average_list is called with just one value, start will have the value 0, which is what we want it to be to start the loop with the first element of the list. In this case, end will again be None, so it will be re-set as before.

numbers = [1.0, 2.0, 5.0]
print '(', numbers, ') =>', average_list(numbers)
print '(', numbers, 1, ') =>', average_list(numbers, 1)
print '(', numbers, 1, 2, ') =>', average_list(numbers, 1, 2)
( [1.0, 2.0, 5.0] ) => 2.66666666667 ( [1.0, 2.0, 5.0] 1 ) => 3.5 ( [1.0, 2.0, 5.0] 1 2 ) => 2.0

Can you draw the stack for these lines of code?

One restriction on default values is that all of the parameters that have default values must come after all of the parameters that don’t. To see why, imagine we were allowed to mix defaulting and non-defaulting parameters like this:

Now let’s go back and figure out what the average of an empty list should be.
Broadly speaking, there are three possibilities:

1. Return 0.0 or some other number.
2. Return some other value, such as None.
3. Treat this as an error,
   i.e.,
   let the divide-by-zero error happen.

Lots of people will use and defend the first choice. (In fact, I have heard people argue that since zero is the average of all possible numbers, it’s the only sensible result). Setting aside objections from mathematical purists, the danger of doing this is that it might mask errors in code explain.

The second option—returning a non-numerical value—is almost always a worse choice, because it complicates the calling code. People want to be able to write:

Returning Values

All of our functions so far have ended with a return statement, and that has been the only return statement they’ve contained. Once again, this doesn’t have to be the case: it is often easier to write functions if they can return from several places, though this can sometimes make them harder to read. Let’s start with a function that calculates the sign of a number:

The psychological term for what’s going on here is
chunking, which refers to the way people group items together in memory. For example, when you look at the five dots on a dice:

picture of dice

what you actually “see” is the X pattern,
and you remember that pattern as a single item rather than remembering five individual dots. Similarly, you remember common words such as “common” as words, not as sequences of letters,
and so on. One of the key differences between experts and novices is that experts are better at chunking: they don’t necessarily have larger short-term memories, but since they recognize a broader repertoire of patterns, they are able to manage more information.
Thus, the more patterns there are in the way a program is laid out,
the easier it will be for people to keep it in their heads.
And as Chase and Simon discuss in their classic paper,
experts actually have a harder time recognizing and remembering things that don’t conform to their repertoire of patterns, which means that if most of a program is laid out one way,
but few bits are organized in some other way, it will be even harder to understand.

This isn’t the only way we can make code hard to understand, of course. For example, here’s another version of our function that doesn’t use an early return.
and only has one conditional branch.
It is probably harder to understand
than either of the previous versions:

def average_list(values):
    result = 0.0
    if len(values) > 0:
        for v in values:
            result += v
        result /= len(values)
    return result

The reason is that the special case isn’t handled explicitly.
Instead,
this function returns 0 for the empty list
because of the code that isn’t executed:
if the list is empty,
the loop doesn’t run,
so the initial value of result
becomes the function’s final value by default.
Keeping track of alternative paths is hard;
noticing and keeping track of paths that aren’t actually in the code is a lot harder,
so many people won’t realize that there is actually a special case at all.

This version of the function is hard to understand for another reason as well.
As we saw in the previous chapter,
most programs use variables in stereotypical ways:
as counters,
as temporaries,
and so on.
code signals that result is an accumulator, but then it’s used directly

One important thing to note about Python is that every function returns something.
If it doesn’t have an explicit return statement,
it returns None.
For example, let’s comment out the last line of our sign function:

def sign(num):
    if num < 0:
        return -1
    if num == 0:
        return 0
# return 1

print -5, '=>', sign(-5)
print 0, '=>', sign(0)
print 241, '=>', sign(241)
-5 => -1 0 => 0 241 => None

The sign of 241 is now None instead of 1,
because when the function is called with a positive value,
neither of the if branches is taken,
and execution “falls off” the end of the function.

Other languages do this differently.
In C,
for example,
trying to use the “result” of a function that doesn’t explicitly return something
is a compilation error—the program can’t even be run.
No matter what the language,
this kind of behavior is one reason why commenting out blocks of code is a bad idea.
It’s all too easy to accidentally get rid of a return statement,
after which your function will silently be telling its caller “no data”.

Aliasing

As we said earlier, did we?
values are copied into parameters when the function is called.
If you recall from the previous chapter,
variables don’t actually store values:
they store references.
If the value is immutable,
copying the value and copying the reference have the same effect.
But if the value is mutable—for example, if it’s a list—then
copying the reference creates an aliases.
This means that the function is working with (or on)
the same list in memory
as its caller.

To explore what this means in practice,
here’s a function that takes a string and a list as parameters,
and appends something to both:

def appender(a_string, a_list):
    a_string += 'turing'
    a_list.append('turing')

And here is some code to set up a pair of variables and call that function.

string_val = 'alan'
list_val = ['alan']
appender(string_val, list_val)
print 'string', string_val
print 'list', list_val
string alan list ['alan', 'turing']

Let’s trace the function’s execution.
Just before the call,
the global frame has two variables
that refer to a string value and a list value:

image before call

The call creates a new stack frame with aliases for those values:

image during call

Since strings are immutable,
the statement a_string += 'turing' creates a new string
and overwrites the value of the function’s local variable a_string
with a reference to it:

image during call

The statement a_list.append('turing'), however,
actually modifies the list that a_list is pointing at,
which is the same thing that list_val in the caller is pointing it:

image during call

When the function returns and the call frame is thrown away,
the new string 'alanturing' is lost,
because the only reference to it was in the stack frame.
The change to the list,
on the other hand,
is kept,
because the function actually modified the list in place:

image after call

need to say something profound here

Libraries

A function is a way to turn a bunch of related statements into a single chunk that can be re-used.
Modularizing code this way eliminates duplication, and makes code easier to read.
A module
or library
(for our purposes, the terms mean the same thing)
does for functions what functions do for statements:
group them together to create more usable chunks.
This hierarchical organization is similar in spirit to that used in biology:
instead of family, genus, and species, we have module, function, and statement.

Every Python file can be used as a module by other programs.
To load a module into a program,
we use the import statement.
For example,
suppose we have created a Python file called halman.py
that defines a single function called threshold:

# halman.py
def threshold(signal):
  return 1.0 / sum(signal)

If we want to call this function in another file,
we write import halman to load the contents of halman.py,
and then call the function as halman.threshold:

import halman
readings = [0.1, 0.4, 0.2]
print 'signal threshold is', halman.threshold(readings)

When we run this program,
it prints the expected answer:

$ python program.py
signal threshold is 1.42857

When a module is imported,
Python executes the statements it contains,
which are usually function defintions.
It then creates an object to store references to all the items defined in that module
and assigns it to a variable with the same name as the module.
For example,
let’s create a file called noisy.py that prints out a message
and then defines NOISE_LEVEL to be 1/3:

# noisy.py
print 'Is this module being loaded?'
NOISE_LEVEL = 1./3.

When we import noisy
the first statement—the print—is executed,
displaying a message on the screen:

>>> import noisy
Is this module being loaded?

Importing the module also defines the variable NOISE_LEVEL,
which we can access as noisy.NOISE_LEVEL:

>>> print noisy.NOISE_LEVEL
0.33333333

Just like a function,
each module is a separate namespace,
so that variable names defined inside a module belong to that module
and don’t collide with variable names defined elsewhere.
When Python sees a reference to a variable,
it looks in the current function call stack frame to find its definition.
If it can’t find it there,
it looks in the module the function was defined in (assuming it was defined in a library).
If it still can’t find it,
it looks in the global namespace belonging to the top-level program as a whole:

picture of name resolution

To see how this works,
let’s create a file called module.py
that defines both a variable called NAME
and a function called func that prints it out:

# module.py
NAME = 'Transylvania'

def func(arg):
  return NAME + ' ' + arg

In our main program,
we also define a variable called NAME,
then import our module.
When we call module.func
it sees the NAME variable that was defined inside the module,
not the one that was defined globally:

>>> NAME = 'Hamunaptra'
>>> import module
>>> print module.func('!!!')
Transylvania !!!

Rules about where and how to look things up might seem pretty arcane,
but it would be practically impossible to write large programs
(or even small ones)
without them.
If every use of the name largest referred to a single unique variable,
then every time we imported a module,
we would have to read it over carefully to make sure that
it wouldn’t inadvertently modify any of the things we were defining in our code.
Restricting lookup to the current function,
its module,
and the top level of the program reduce the chances of accidental collision.
It also makes it easier for people to understand code,
since there are only three places where the variables used on a particular line might be,
two of which (the containing function and the file it’s in)
are guaranteed to be nearby.

Standard Libraries

The real power of a language is in its modules:
they are the distilled wisdom (and effort)
of all the programmers who have come before us.
Python’s standard library contains over a hundred modules,
and the fastest way to become a more productive programmer
is to learn what these contain.
One of the most useful is math,
which defines sqrt for square roots,
hypot for calculating x²+y²,
and values for e and π that are as accurate as the machine can make them.

>>> import math
>>> print math.sqrt(2)
1.4142135623730951
>>> print math.hypot(2, 3)  # sqrt(x**2 + y**2)
3.6055512754639891
>>> print math.e, math.pi   # as accurate as possible
2.7182818284590451 3.1415926535897931

Since math.sqrt is a handful to type,
and sqrt is probably not ambiguous,
Python provides a few convenient alternatives for doing imports.
For example, we can import specific functions from a library and then call them directly,
rather than using the modulename.functionname syntax:

>>> from math import sqrt
>>> sqrt(3)
1.7320508075688772

We can also import a function under a different name,
so that if two modules define functions with the same name,
we can give one or the other a different name when we want to use them together:

>>> from math import hypot as euclid
>>> euclid(3, 4)
5.0

We can also use import *
to bring everything in the module into the current namespace at once.
This has the same effect as using from module import a,
from module import b, and so on for every name in the module:

>>> from math import *
>>> sin(pi)
1.2246063538223773e-16

import * is almost always a bad idea.
If someone adds a new function or variable to the next version of the module,
your import * could silently overwrite something that you have written,
or are importing from somewhere else.
Bugs like this can be extremely hard to find,
since nothing seemed to change in your program.

While the math library is useful,
the sys library is even more so.
Once it’s imported, we can find out exactly what version of Python we’re using,
what operating system we’re running on,
how large integers are,
and a few other things:

>>> import sys
>>> print sys.version
2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)]
>>> print sys.platform
win32
>>> print sys.maxint
2147483647
>>> print sys.path
['', 'C:\\WINDOWS\\system32\\python27.zip', 'C:\\Python27\\DLLs', 'C:\\Python27\\lib', 'C:\\Python27\\lib\\plat-win', 'C:\\Python27', 'C:\\Python27\\lib\\site-packages']

The most commonly-used element of sys, though, is sys.argv,
which holds a list of the command-line arguments
used to run the program.
The name of the script itself is in sys.argv[0];
all the other arguments are put in sys.argv[1], sys.argv[2], and so on.
For example, here’s a program that does nothing except print out its command-line arguments:

# echo.py
import sys
for i in range(len(sys.argv)):
  print i, '"' + sys.argv[i] + '"'

If it is run without any arguments,
it reports that sys.argv[0] is echo.py:

$ python echo.py
0 echo.py

When it is run with arguments, though, it displays those as well:

$ python echo.py first second
0 echo.py
1 first
2 second

sys also creates variables to connect programs to standard I/O channels.
sys.stdin is standard input,
which is usually connected to the keyboard.
sys.stdout is standard output,
which by default is connected to the screen,
while sys.stderr is standard error,
which is also usually connected to the screen.
Please see the chapter on the Unix shell
for more information on what these are for and how to use them.

Here’s a typical example of how these variables are used together:

# count.py
import sys
if len(sys.argv) == 1:
  count_lines(sys.stdin)
else:
  rd = open(sys.argv[1], 'r')
  count_lines(rd)
  rd.close()

This program looks at sys.argv to see if it was called with a filename as an argument or not.
If there were no arguments,
then sys.argv will only hold the name of the program,
and its length will be 1.
In that case, the program reads data from standard input:

$ python count.py < a.txt
48

Otherwise,
the program assumes its first command-line argument is the name of an input file,
opens it,
and reads from it instead:

$ python count.py b.txt
227

Here’s a more polite way to write the program we just created:

'''Count lines in files.  If no filename arguments given,
read from standard input.'''

import sys

def count_lines(reader):
  '''Return number of lines in text read from reader.'''
  return len(reader.readlines())

if __name__ == '__main__':
  if len(sys.argv) == 1:
    print count_lines(sys.stdin)
  else:
    r = open(sys.argv[1], 'r')
    print count_lines(r)
    r.close()

The two significant changes are
the strings at the start of the module and of the function count_lines,
and the funny-looking line that compares __name__ to '__main__'.
Let’s look at them in that order.

To help us find our way around libraries,
Python provides a help function.
If math has been imported,
the call help(math) prints out the documentation embedded in the math library:

>>> import math
>>> help(math)
Help on module math: NAME math FILE /usr/lib/python2.5/lib-dynload/math.so MODULE DOCS http://www.python.org/doc/current/lib/module-math.html DESCRIPTION This module is always available. It provides access to the mathematical functions defined by the C standard. FUNCTIONS acos(...) acos(x) Return the arc cosine (measured in radians) of x. ⋮ ⋮ ⋮

Here’s how this works.
If the first thing in a module or function other than blank lines or comments is a string,
and that string isn’t assigned to a variable,
Python saves it as the documentation string,
or docstring,
for that module or function.
These docstrings are what online (and offline) help display.
For example, let’s create a file adder.py with a single function add,
and write docstrings for both the module and the function:

# adder.py
'''Addition utilities.'''

def add(a, b):
  '''Add arguments.'''
  return a+b

If we import adder,
help(adder) prints out all of its docstrings,
i.e., the documentation for the module itself and for all of its functions:

>>> import adder
>>> help(adder)
NAME adder - Addition utilities. FUNCTIONS add(a, b) Add arguments.

We can also be more selective,
and only display the help for a particular function instead:

>>> help(adder.add)
add(a, b) Add arguments.

The second part of our “more polite” program was that funny if statement.
The trick here is that when Python reads in a file,
it assigns a value to a special top-level variable called __name__
(with two underscores before and after).
If the file is being run as the main program,
__name__ is assigned the string '__main__'
(again with two underscores before and after).
If the file is being loaded as a module by some other program,
though, Python assigns the module’s name to the variable __name__ instead.

Now,
suppose that the file contains some definitions and then the conditional statement
if __name__ == '__main__'.
The definitions will always be executed,
but the code inside the conditional will only run if the file is the main program.
Put another way,
the statements inside the conditional will not be run
if the file is being loaded as a library by some other program.

That’s pretty confusing,
so here’s an example to show how it works.
The file stats.py defines a function average,
and then runs three simple tests—but
only if __name__ has the value '__main__':

# stats.py
'''Useful statistical tools.'''

def average(values):
  '''Return average of values or None if no data.'''
  if values:
    return sum(values) / len(values)
  else:
    return None

if __name__ == '__main__':
  print 'test 1 should be None:', average([])
  print 'test 2 should be 1:', average([1])
  print 'test 3 should be 2:', average([1, 2, 3])

If we import this file int an interactive session,
it doesn’t produce any output,
because stats.__name__ has been assigned the value 'stats':

>>> import stats
>>> print stats.__name__
stats

If we run this file directly, though,
that same __name__ variable will be assigned the value '__main__',
so the test will be run:

$ python stats.py
test 1 should be None: None test 2 should be 1: 1 test 3 should be 2: 2

This is another common design pattern in Python:
group related functions into a module,
then put some tests for those functions in the same module
under if __name__ == '__main__',
so that if the module is run as the main program,
it will check itself.

Functions as Objects

As we’ve seen previously,
an integer is just 32 or 64 bits of data that a variable can refer to,
while a string is just a sequence of bytes that a variable can also refer to.
Functions are just another sequence of bytes—ones that happen to represent instructions.
Variables can refer to them too, just as they can refer to any other data.

This insight—the fact that code is just another kind of data,
and can be manipulated like integers or strings—is
one of the most useful and powerful in all computing.
To understand why,
let’s have a closer look at what actually happens when we define a function:

def threshold(signal):
    return 1.0 / sum(signal)

These two lines tell Python that threshold is
a function that returns one over the sum of the values in signal.
When we define it,
Python translates the statements in the function into a blob of bytes,
then creates a variable called threshold and makes it point at that blob:

picture of function being defined

This is not really any different from assigning the string 'alan turing'
to the variable name:

add picture of string assigned to variable

the only difference is what’s in the memory the variable points to.

If threshold is just a reference to something in memory,
we should be able to assign that reference to another variable.
Sure enough,
we can:

picture of aliased function

All we have done is make t point to the same data as threshold,
i.e, create an alias for the function.
To prove this is so, let’s try calling t:

t = threshold
print t([0.1, 0.4, 0.2])
1.42857

The result is exactly what we would get
if we called threshold with the same parameters,
because t and threshold are the same function.

If a function is just data,
can we put a reference to it in a list?
Let’s define two functions, area and circumference,
each of which takes a circle’s radius as a parameter and returns the appropriate value:

def area(r):
    return pi * r * r

def circumference(r):
    return 2 * pi * r

Once those functions are defined, we can put them into a list:

funcs = [area, circumference]

Of course, what we really mean is
“put references to the functions in a list”:

picture of list of functions

We can now loop through the functions in the list,
calling each in turn.
Sure enough,
the output is what we would get if we called area
and then circumference:

for f in funcs:
    print f(1.0)
3.14159 6.28318

Let’s go a little further.
Instead of storing a reference to a function in a list,
let’s pass that reference into another function,
just as we would pass a reference to an integer, a string, or a list.
Here’s a function called call_it that takes two parameters:
a reference to some other function, and some other value.

def call_it(func, value):
    return func(value)

All call_it does is call that other function with the given value as a parameter.
Let’s test it:

print call_it(area, 1.0)
3.14159

print call_it(circumference, 1.0)
6.28318

Now it’s time for the payoff.
Here’s a function called do_all
that applies some function—anything at all that takes one argument—to
each value in a list, and returns a list of the results:

def do_all(func, values):
    result = []
    for v in values:
        temp = func(v)
        result.append(temp)
    return result

If we call do_all with area and a list of numbers,
we get what we would get if we called area directly on each number in turn:

print do_all(area, [1.0, 2.0, 3.0])
[3.14159, 12.56636, 28.27431]

And if we define a function to “slim down” strings of text
by throwing away their first and last characters,
we can apply it to every string in a list,
without having to write another copy of the code that loops through the list,
calls the function,
and concatenates the results:

def slim(text):
    return text[1:-1]

print do_all(slim, ['abc', 'defgh'])
b efg

Functions that operate on other functions are called
higher-order functions.
They’re common in mathematics:
average, for example,
is a function that takes some other function and a range as arguments.
In programming, higher-order functions allow us to re-use control flow rather than rewriting it.
Putting it another way,
they allow us to eliminate redundancy in our code,
so that we only have to write and test things once.

As another example,
let’s look at combine_values,
which takes a function and a list of values as parameters,
and combines the values in the list using the function provided:

def combine_values(func, values):
    current = values[0]
    for i in range(1, len(values)):
        current = func(current, v)
    return current

To show how this works,
let’s define add and mul to add and multiply values:

def add(x, y):
    return x + y

def mul(x, y):
    return x * y

If we combine 1, 3, and 5 with add, we get their sum, 9:

print combine_values(add, [1, 3, 5])
9

If we combine the same values with mul, we get their product, 15:

print combine_values(mul, [1, 3, 5])
15

This same higher-order function combine_values could concatenate lists of strings, too,
or multiply several matrices together, or whatever else we wanted,
without us having to write or test the loop ever again.
Without higher-order functions,
we would have to write one function for each combination of data structure and operation,
i.e.,
one function to add numbers,
another to concatenate strings,
a third to sum matrices,
and so on.
With higher-order functions, on the other hand,
we only write one function for each basic operation,
and one function for each kind of data structure.
Since A plus B is usually a lot smaller than A times B,
this saves us coding, testing, and debugging.

Several higher-order functions are actually built in to Python.
One is filter,
which constructs a new list containing all the values in an original list
for which some function is true:

def positive(x):
    return x > 0

print filter(positive, [-5, 3, -2, 9, 0])
[3, 9]

Another is map,
which applies a function to every element of a list and returns a list of results:

def bump(x):
    return x + 10

print map(bump, [-5, 3, -2, 9, 0])
[5, 13, 8, 19, 10]

And then there’s reduce,
which combines values using a binary function,
returning a single value as a result:

def add(x, y):
    return x + y

print reduce(add, [-5, 3, -2, 9, 0])
5

Combining all of these is a very powerful way to do a lot of computation
with very little typing:

print reduce(add, map(bump, filter(positive, [-5, 3, -2, 9, 0])))
32

Reading from the inside out, we have:

1. filtered the list, keeping only the positive values,
2. bumped them up by 10, and
3. summed the results.

Written out, this is:

total = 0
for val in [-5, 3, -2, 9, 0]:
    if val > 0:
        total += (val + 10)
print total

Say something here about the higher-level logic being lost in the details.
But there’s an efficiency tradeoff.

Summing Up

To close off this chapter,
let’s try to answer a frequently-asked question:
when should we write functions?
And what should we put in them?
The answer depends on the fact that
human short-term memory can only hold a few things at a time.
If we try to remember more than a double handful of unrelated bits of information
for more than a few seconds,
they become jumbled and we start making mistakes.
In particular,
if someone has to keep several dozen things straight in their mind
in order to understand a piece of code,
that code is too long.

Functions are a way to divide code up into more comprehensible pieces:
essentially,
to replace several pieces of information with one to make the whole easier to understand.
Functions are therefore not just about eliminating redundancy:
they are worth writing even if they’re only called once.

Let’s consider an example:

for x in range(1, GRID_WIDTH-1):
  for y in range(1, GRID_HEIGHT-1):
    if (density[x-1][y] > density_threshold) or \
       (density[x+1][y] > density_threshold):
      if (flow[x][y-1] < flow_threshold) or\
         (flow[x][y+1] < flow_threshold):
        temp = (density[x-1][y] + density[x+1][y]) / 2
        if abs(temp - density[x][y]) > update_threshold:
          density[x][y] = temp

This code uses meaningful variable names,
and is well structured,
but it’s still a lot to digest in one go.
Let’s start by replacing the loop bounds with function calls
that give us a bit more context:

for x in grid_interior(GRID_WIDTH):
  for y in grid_interior(GRID_HEIGHT):
    if (density[x-1][y] > density_threshold) or \
       (density[x+1][y] > density_threshold):
      if (flow[x][y-1] < flow_threshold) or\
         (flow[x][y+1] < flow_threshold):
        temp = (density[x-1][y] + density[x+1][y]) / 2
        if abs(temp - density[x][y]) > update_threshold:
          density[x][y] = temp

grid_interior(num) might just return range(1, num-1),
but try reading the first two lines of this code aloud,
and then the first two lines of what it replaced.
This version is easier to understand.

Now let’s replace those two conditionals with function calls as well:

for x in grid_interior(GRID_WIDTH):
  for y in grid_interior(GRID_HEIGHT):
    if density_exceeds(density, x, y, density_threshold):
      if flow_exceeds(flow, x, y, flow_threshold):
        temp = (density[x-1][y] + density[x+1][y]) / 2
        if abs(temp - density[x][y]) > tolerance:
          density[x][y] = temp

Again, we’ve reduced the number of things our eyes have to scan,
and provided more information about what we’re actually doing.

Finally, let’s create and call a function
to handle updates to our data structure:

for x in grid_interior(GRID_WIDTH):
  for y in grid_interior(GRID_HEIGHT):
    if density_exceeds(density, x, y, density_threshold):
      if flow_exceeds(flow, x, y, flow_threshold):
        update_on_tolerance(density, x, y, tolerance)

Our original nine lines have become five,
and those five are all at the same mental level.
It’s hard to pin down exactly what that phrase means,
but most programmers would agree that
the first version mixed high-level ideas about boundaries and update conditions
with low-level details of grid access and cell value comparisons.
In contrast,
this version only has the high-level stuff;
the low-level implementation details are hidden in those functions.

A conscientious programmer who wrote the code we started with
would go back and refactor it
to turn it into something like our final version
before committing it to version control.
If she did this often enough,
she would eventually find herself writing the final version first,
just as mathematicians find themselves skipping more and more “obvious” steps
as they do more proofs.
When we see someone “just writing” something elegant,
the odds are good that they have spent time rewriting their own poor code,
and in doing so,
turned conscious decision into unconscious action.

Finally,
let’s step back and ask, “What is programming anyway?”
Novices usually think that it means writing instructions for a computer.
But more experienced programmers think of it as creating and combining abstractions.
When we’re programming,
our goal is to spot a pattern,
like “combine all elements of a list using a binary function,”
and then write it down once,
as clearly as possible,
so that we can build more patterns on top of it.

Be cautious, though: the limits of human short-term memory still apply.
If you pile too many abstractions on top of one another,
it can be difficult to figure out what the end result actually does.