 |
Hello, and welcome to the sixth episode of the Software Carpentry lecture on testing. In this episode, we’ll have a closer look at how to set up fixtures to run tests on. |
 |
Let’s go back to the fields of Saskatchewan. |
 |
We’re trying to find places where photographs of those fields overlap. |
 |
Each photograph contains one or more rectangles. |
 |
In programming terms, this means that a photo is a collection of rectangles, such as a set or a list. |
 |
What we really want to do is find all the overlaps between the rectangles in two collections. |
 |
For example, if the first photo contains the three fields shown in yellow, and the second contains the two fields shown in green, our output should be the five rectangles shown in blue. |
 |
We’ve already tested a function called overlap_rect that finds the overlap between two rectangles. |
 |
We’re confident it works, so we now want to test overlap_photo. |
 |
We don’t actually need to know how it works, but we imagine that its implementation looks something like this: |
 |
The two loops get rectangles from the first and second photo respectively, so that we’re comparing all against all. |
 |
Inside the loop, we compare those rectangles and save their overlap if it’s non-empty. |
 |
Here’s our first test: a photo containing only the unit square compared with itself. The output should be just the unit square. |
 |
And here’s how we turn that test into code. |
 |
Six lines of code: that’s not too bad. |
 |
OK, let’s write a second test: the unit square against a checkerboard pattern containing four squares. Again, the output should contain only the unit square. |
 |
Again, it’s only six lines of code… |
 |
…but those six lines are harder to read. |
 |
We could introduce a variable called unit so that we don’t write down the unit square’s coordinates twice, but it doesn’t really help much. |
 |
Here’s our third test: a short and wide rectangle against the checkerboard, with two squares as output. |
 |
Again, it’s only six lines of code… |
 |
…but it isn’t particularly easy to read… |
 |
…and we can now see a new problem: there’s too much duplicated code. We’ve now defined the unit square four or five times, and the checkerboard twice. Duplication is almost always a sign that a program could be simplified; the only question is how. |
 |
In the case of testing, the answer is simple: let’s create our fixtures outside the tests, so that we only have to define them once, and many different tests can share them. |
 |
(Remember, a fixture is something that we run a test on—in this case, our photos.) |
 |
This is a common need, so like other testing libraries, Nose can help us out. If a file contains a function called setup, Nose will automatically run that function before it runs any of the tests in the file. |
 |
Here’s an example showing how it works. |
 |
If we were actually testing, setup would define the fixtures our tests used, but we’ll have it print to standard error so that we can see when it runs. |
 |
Similarly, test_1 and test_2 would use those fixtures to do some testing, but we’ll have them print as well. |
 |
When we run this file with the nosetests command, it produces this output. |
 |
The stuff shown in blue is Nose’s regular output; our print statements are interleaved with it, which makes things a bit hard to read. |
 |
This is what Nose’s regular output would look like without our print statements. |
 |
As you can see, Nose ran setup once at the start… |
 |
…and then ran our two tests (in some arbitrary order—remember, Nose gets to choose). |
 |
All right, let’s write a setup function to create some fixtures for testing photo overlap. |
 |
Here’s the code. |
 |
We start by creating a global variable called Photos to hold our fixtures. We use a global variable, outside any particular function, so that both setup and our individual tests can see it. |
 |
Inside setup, we create sets of rectangles and store them in Photos. |
 |
Now let’s use those fixtures in some tests. |
 |
Here’s the overlap of the unit square with itself. |
 |
And here’s the overlap of our short and wide rectangle with the checkerboard. |
 |
We don’t have to put all our fixtures in one variable, of course—we could create a bunch of global variables, and store one fixture in each. |
 |
Here’s the code to do that. Notice that we have to assign each global variables some value just to create it. |
 |
Which you use is a matter of personal taste and project style. |
 |
Of course, in this particular case you don’t actually need a setup function at all—you can assign fixtures to Unit and Short_And_Wide when they’re first defined. |
 |
However, when you’re working with more complicated programs, creating fixtures can require many lines of code, and calls to helper functions, and bundling all of that into a setup function is tidier. |
 |
And even when you can do everything in one go, there’s still another good reason to use a setup function. What happens if some or all of your tests modify the fixtures they run on? |
 |
For example, suppose we’re testing a function called photo_crop that removes rectangles lying completely outside some cropping window. |
 |
Since our tests modify our fixtures, we can’t use the same fixtures over and over in separate tests—if we did, a bug in one place would contaminate the results of tests that were run later. |
 |
The solution is to re-create the fixtures for each test, i.e., to run setup over again just before each test function. |
 |
There are several ways to do this in Nose (and in other libraries), but the simplest to type is to use a decorator. |
 |
Here’s what the code looks like. |
 |
We import the decorator, called with_setup, from the Nose library, just as if it were a function… |
 |
…because it actually is a function, just one that behaves in a special way. |
 |
We then put @decorator—in this case, @with_setup—right before the definition of each function that we want to apply it to. This decorator takes an argument—the name of the setup function to run before the test—so we pass that to the decorator just like an argument to a function call. |
 |
Thanks to a bit of magic that we won’t go into here, doing this tells Nose to run the setup_each function right before test_1, and again before test_2. If we wanted to run different setup functions before the two tests, we would simply pass different function names to with_setup. |
 |
Here’s the output when we run this program with nosetests. |
 |
Once again, the standard Nose output in blue shows that two tests were run successfully. |
 |
If we look at the output from our own print statements, we can see that Nose ran setup_each right before it ran test_1… |
 |
…and ran it again before test_2, just as we wanted. |
 |
Here’s an example showing setup per test that actually does some testing. (We’ve left out the definition of create_fixtures because you’ve probably seen enough rectangles by now. |
 |
When we run this program, Nose calls create_fixtures to create the first copy of the checkerboard fixture… |
 |
…then runs test_crop_unit, which modifies that checkerboard. |
 |
Nose then runs create_fixtures again, assigning a fresh photo to checkerboard… |
 |
…and then runs test_crop_keep_everything, which modifies that copy in turn, and so on. |
 |
Re-running our setup function over and over again does waste a few microseconds of the computer’s time. |
 |
But that’s much less valuable than any of yours. And if you ever decide that creating a thousand fixtures, but only using one or two, really is too expensive, you can always break fixture creation up into several functions, and use the decorator to call only the ones you need for a particular test. |
 |
Decorators might seem like magic, but they aren’t. |
 |
They do, however, use some ideas that are outside the scope of this course. |
 |
You don’t have to understand how they work in order to use them… |
 |
…just as you don’t have to understand how Nose finds tests in files, or files that contain tests. |
 |
All you really have to understand is: |
 |
…what the @with_setup decorator does, and… |
 |
…when and why to use it. |
 |
Thank you for listening. |
What’s the advantage of using the decorator instead of directly calling the fixture function from the test?
@Jochen Good question — I think the answers are “readability” and “predictability”:
1. You can see at a glance that setup is being done, without reading the internals of the test function. In a way, this makes calling the setup function part of the test function’s contract with the outside world, rather than something it just happens to do.
2. Putting the call to setup under Nose’s control ensures that the call is made exactly the same way for every test function; there are no “local variations” to confuse us.
Is this enough to justify introducing a new concept (decorators) to people who are still relatively junior programmers?
You’re pulling a fast one with decorators. What it looks like is that setup_each is being called before the test function. Only the astute will realize this isn’t the case, since nose is calling only test_* functions and is not executing the whole file top to bottom (they’ve seen Nose for what, 10 minutes now?). The notion that a decorator reaches into the function and modifies it is totally foreign even to most experienced scientific programmers (maybe even to most programmers, period). So, they will gin up other – wrong – explanations for what a decorator does, since you didn’t supply one and they are, after all, scientists trained to gin up explanations for unknown things they encounter. You’ve given them magic, when the whole point of a course like this is to dispel magic and replace it with understanding.
So if you are going to use the decorator, you need to explain at least in general terms what it does and why you care to use it. A quick Google does not turn up a simple explanation of the Python decorator in the first page of hits. Also, this concept does not generalize to most other languages.
Finally, I don’t buy your replies to Jochen. First, you never explained what with_setup does, so how exactly is Nose controlling it and why would it want to? Second, the source code for the function is right there. Swap the decorator and the def line and you have exactly the same amount of reading, only you can eliminate an entire concept and just call the function – MUCH shorter and simpler to all but experienced Python programmers. You have a minor point with the public interface of the test function, but at what price for the complexity of your teaching example?
–jh–
I have to admit, I share Jochen’s problem with understanding the advantage to decorators. It calls the setup function that refreshes the fixture value, but I don’t get why that’s any better than just calling the function normally.
I notice that that dictionaries are used even though they haven’t been explained yet.
I feel like either one should explain the decorator concept in more detail or else simply drop it from this episode. My very very cursory glance at Wikipedia tells me that a decorator is a function that modifies another function?