Literate Programming
Last week’s post about the tuple space programming model was so popular that I thought readers might enjoy a discussion of another beautiful idea that failed: literate programming. Like Lisp and other toenail-based languages, it inspires a kind of passion in its fans that is normally reserved for gods, sports teams, and angsty rock bands. And, like them, it leaves everyone else wondering what the big deal is.
Literate programming was invented by Donald Knuth (one of the few real geniuses ever to grace computer science) as a way of making programs easier to understand. His idea was that the code and the documentation should be a single document, written in a free-flowing mixture of Pascal and TeX, C and LaTeX, or more generally, a text markup language and a programming language. Functions, classes, modules, and other things could be introduced and explained in whatever order made sense for human readers. One tool would extract and format the text-y bits to create documentation, while another would extract and compile the code-y bits to produce the runnable program.
It’s a great idea, and for about six months in the late 1980s, I was convinced it was the future of programming. I could use δ as a variable! I could call a function for calculating sums Σ(…)! My explanation of what the code was doing, and the code itself, were interleaved, so that whenever I changed one, I would naturally change the other, so that they never fell out of step! And with a bit of tweaking, I could produce a catalog of functions (this was before I started doing object-oriented programming), or present exactly the same content in breadth-first order, the way it was executed (which was usually easier for newcomers to understand). Cool!
But then I had to maintain a large program (20K lines) written with literate tools, and its shortcomings started to become apparent. First and foremost, I couldn’t run a debugger on my source code: instead, my workflow was:
- “compile” the stuff I typed in—the stuff that was in my head—to produce tangled C;
- compile and link that C to produce a runnable program;
- run that program inside a debugger to track down the error;
- untangle the code in my head to figure out where the buggy line(s) had come from;
- edit the literate source to fix the problem; and
- go around the loop again.
After a while, I was pretty good at guessing which lines of my source were responsible for which lines of C, but the more use I made of LP’s capabilities, the more difficult the reverse translation became. It was also a significant barrier to entry for other people: they had to build a fairly robust mental model of the double compilation process in order to move beyond “guess and hack” debugging, whereas with pure C or Fortran, they could simply fire up the debugger and step through the stuff they had just typed in.
I also realized after a while that the “beautiful documentation” promise of LP was less important than it first appeared. In my experience, programmers look at two things: API documentation and the source code itself. Explanations of the code weren’t actually that useful: if the programmer was treating the code as a black box, she didn’t want to know how it worked, and when she needed to know, she probably needed to see the actual source to understand exactly what was going on (usually in order to debug it, or debug her calls to it). The only role in between where LP was useful lay in giving an architectural overview of how things fit together, but:
- that was something people only really needed once (though when they needed it, they really needed it), and
- that level of explanation is really hard to write—exactly as hard, in fact, as writing a good textbook or tutorial, and we all know how rare those are.
So I moved on, and so did most other fans of LP. But then Java happened, and history repeated itself, not as tragedy, but as farce. The first time I saw Javadoc, I thought it looked like it had been invented by someone who’d heard about literate programming in a pub, but had never actually seen it. I later realized that was unfair: Javadoc was the closest thing to LP that Java’s inventors thought they could get away with, and it actually did lead more programmers to write more documentation than they ever had before. But saints and small mercies, look at what it doesn’t do:
- There’s no checking: you can document parameters that don’t exist, or mis-document the types and meanings of parameters that do.
- You can only put Javadoc at the start of a class or method, rather than next to the tricky bit of code in the middle of the method that implements the core algorithm. (Though to be fair, if the method is long enough that this is a problem, it should probably be refactored into several smaller methods.)
- There’s no logical place for higher-level (architectural) documentation: Javadoc really is designed for describing the lowest (API) level of code.
- You have to type and view HTML tags.
That last point might seem a small one, but it’s the key to understanding what’s actually wrong with this model. Think about it: everyone who’s writing Java has, on their desktop, a WYSIWYG tool such as Microsoft Word that renders italics as italics, links as links, tables as tables, and so on. When they start writing code, though, they have to type <strong>IMPORTANT</strong> to emphasize a word, or something as barbaric as:
<table border="1">
<tr>
<td colspan="2" rowspan="2" align="center">Result</td>
<td colspan="2" align="center">left input α</td>
</tr>
<tr>
<td>>=0</td>
<td><0</td>
</tr>
<tr>
<td rowspan="2" align="center">right<br/>input<br/>β</td>
<td>>=0</td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<td><0</td>
<td>0</td>
<td>-1</td>
</tr>
</table>
to get something that anyone else in the 1990s (never mind the 21st Century) would create with one menu selection:
| Result | left input α | ||
| >=0 | <0 | ||
| right input β |
>=0 | 1 | 0 |
| <0 | 0 | -1 | |
And don’t get me started on diagrams: every decent programming textbook has block-and-arrow pictures of linked lists, dataflow diagrams, and what-not, because these aid understanding. Not source code, though; the closest you can come is to create a diagram using some other tool, save it as a JPEG or PNG, put it somewhere that you hope it won’t be misplaced, and include a link to it in your source code. The picture itself won’t be visible to people looking at your code, of course—they’ll have to decode the link and open the picture manually, assuming of course that it hasn’t been misplaced—but hey, if their intellects are so weak that they need pictures, well, what are they doing looking at code anyway?
The tragedy (or irony) is that we know how to solve this problem, because we’ve been solving it for other people for almost forty years. Electrical engineers and architects don’t use Microsoft Paint to draw circuit diagrams and blueprints; instead, they use CAD tools that:
- store a logical model of the circuit or building in a form that’s easy for programs to manipulate;
- display views of that model that are easy for human beings to understand and manipulate; and
- constrain what people can do to the model via those views.
What’s the difference?
- In an architectural CAD package, I can’t put a door in the middle of nowhere: it has to be in a wall of some kind. In Emacs or Eclipse, on the other hand, I can type any gibberish I want into a Java file, or write Javadoc about an integer parameter called
thresholdwhen in fact I have two floating point parameters calledminandmax. - That CAD package will let me show, hide, or style bits of the model: I can see plumbing and electrical, but not air vents, or windows and doors but not floors, and so on, and I can see those things in several different ways. When I’m looking at source code, I can’t even see my Javadoc rendered in place.
The root of the problem is that programmers—including the ones who design programming languages—still insist that programs have to be stored as sequences of characters, and that that’s all that will be stored. Even new languages created by really smart people stay stuck in this sandpit. Why? Because that’s all that compilers and debuggers and other tools understand? Well, you’re writing new ones anyway, aren’t you?
No, I’m convinced that the real reason is that plain old text is the only common denominator that programmers’ editors understand. Most programmers will change language, operating system, nationality, even gender before they’ll change editors. (Hell, I’m typing this in Emacs, rather than using a WYSIWYG HTML editor—how sad is that?) Most therefore assume, probably correctly, that if a language requires people to give up the years they have spent learning what Ctrl-Alt-Shift-Leftfoot-J does, they will ignore it. They’ll continue to build level editors for computer games, but use a souped-up typewriter to do it.
Sooner or later, though, one of the many multi-modal CAD tools for programmers that people have built over the years will take off, just as object-oriented programming and hypertext eventually did after gestating in obscurity for years. I’ve argued before that the most likely candidate is a proprietary programming environment like Visual Basic or MATLAB, where a a single vendor with a more or less captive audience can roll out a whole toolchain at once without worrying arguing it through standards committees. I’m not holding my breath, though; while the recent surge of interest in “innovative” programming languages is welcome, it feels to me like everyone is trying to design better balloons rather than saying, “Hey, birds are heavier than air—why don’t we give that a try?”
Mathematica has some of the properties you write about here. I haven’t seen any survey data, but my impression is that it has a small userbase compared to, say, Matlab. Why do you think Mathematica isn’t more popular?
Matlab’s success is akin to that of Java in that it provides a massive “JDK” which people find useful. However, Matlab is poorly designed from a language perspective and lacks basic consistency present in nearly every modern language (save perhaps PHP). For example, that ‘ones(1); ans(1)’ returns one but ‘ones(1)(1)’ is a syntax error boggles my mind. Strangely, this doesn’t bug Matlab’s core user as their target audience is often exposed to Matlab prior to them expecting such consistency from a language. Being a bit more CS-ish, the Octave folks have corrected such issues where possible and, for example, ‘ones(1)(1)’ works beautifully in Octave.
If you want mathy symbols, Fortress is interesting. I’ve never used it.
Literate Haskell is an interesting fallout from the literate programming movement– it is a literate capability purely built into the Haskell language. I believe it gets used.
Doxygen addresses many of the shortcomings of Javadoc that you mention.
LabView provides a fairly populate graphical programming language (formerly called “G”, I think). It started out as a way to get data off devices and resembles a circult with wires. All ordering of operations is data-driven. At something beyond (by my estimate) 2,000 lines of regular program functionality, figuring out what’s happening becomes (by my estimate) terribly difficult.
Mathematica isn’t more popular because Wolfram charges too much for it. Ever priced a single seat, non-academic license? Unless most individuals already know they absolutely need Mathematica, the license costs are high enough that they discourage casual Mathematica dabbling.
Javadoc does have a place for high-level docs, you can add documents to the package level, and if you read most of Sun’s packages you’ll see quite a lot of docs there. It still has to be HTML, of course.
One alternative to HTML is Markdown (or more usefully, Multimarkdown), which allows near-plain-text documents to generate rich markup like HTML. Most projects on Bitbucket or Github have a README.md now, because Markdown is so much more useful than HTML.
It would not be hard to make a Javadoc replacement that used Markdown in comments, and even gathered comments inside the method.
Markdown, HTML, hieroglyphics — they’re all still plain ASCII text that have to be compiled to be viewed as they’re intended, in an age when everyone who isn’t a a nerd can see what they’re writing as they’re writing it, insert diagrams in situ, draw tables, etc. To continue the flying analogy from the original article, wiki vs. HTML is like asking “hydrogen or helium?”
Really interesting post, but not sure I agree with your last point. I have to agree with The Pragmatic Programmer: Keep Knowledge in Plain Text. “Plain text won’t become obsolete. It helps leverage your work and simplifies debugging and testing.”
I don’t know exactly what you’re suggesting, but it feels like you think we should move away from plain text-only sources. Like they say above though, plain text won’t become obsolete. And, really, it’s the only format that won’t, so I think it’s got a lot going for it.
I may have completely missed your point though, so if I did, apologies
You haven’t missed your point, but you have reinforced it
Yes, text will always be with us: does that mean we should use text for CAD models? Or (closer to home) only use text in textbooks, never diagrams or tables?
Visual Studio .NET addresses some of the gripes with javadoc; it at least checks to make sure you aren’t documenting parameters that don’t exist. Unfortunately that’s the only point it addresses; the only reasonable place for high-level documentation is on the class itself and it’s not as useful as a standalone topic. Plus you have to use XML; it’s great to have more lines of tag soup than the method takes up! Even worse, there isn’t a mature tool for processing this documentation. Once there was NDoc, but the developer had a meltdown and now we’re stuck with Sandcastle, a perpetually-in-alpha tool by Microsoft maintained by one (maybe two) developers that requires end-user customization to provide the kinds of things we want.
I think you’re right that graphical programming environments can fix a lot of these problems (I’m obligated to ask if you’ve looked at LabVIEW!) but I’ve never been able to grok graphical programming like textual programming. Perhaps the ideal editor would graphically display dependencies and communication between entities but let you edit the behavior of the entities via text? Now I’m just speculating.
I’ve never met a purely graphical programming environment that I liked, but I’ve also never seen a wordless textbook or manual that I liked either. When they’re allowed to do anything, people mix the two in a much wider variety of ways than current programming tools allow; (almost) everyone else’s authoring tools allow this mix as well, so why not ours?
Bravo!
I come from a mixed world of eng/physics where the CAD systems are reality based. In turn, those systems are based on thousands of years of learning how to describe reality, whether a bronze age alloy or a transistor. No surprise we are struggling with algorithms. Systems engineering began in the 60′s with minuteman missiles and was honed by NASA and the moon program.
I was part of a team that used a pre IBM version of STATEMATE ( a visual CAD/code system ) to uncover major errors in the the auto pilot system requirements doc. We, with little experience in auto pilots, could see the errors! Of course, the real requirements doc was in the grey heads of the senior system engineers. Better to have both views.
I highly recommend, Causality, by Judea Pearl, a robotics & cognitive science guy at UCLA. He uses the same CAD engineering ideas to try to understand how to teach a robot what we know and how it can learn by experience.
I’m exploring a combination of LabVIEW, a visual instrumentation CAD system, Mathematica, and Bayesian nets to model public policy debates. Hopeless right?
yes I’m a part time lecturer, so I get a break on Mathematica prices, but the home version I think has full capability, but limited computational power ( ~ $300 ) for dabbling.
“In my experience, programmers look at two things: API documentation and the source code itself.” This is why I propose literate testing instead of programming. See http://arrenbrecht.ch/testing/.
Any Smalltalk environment (except GST) is already much closer to what you describe than any of the mainstream IDEs.
Can you give me an example (e.g., a screenshot) of a Smalltalk IDE that allows me to embed pictures directly in source code? Or draw decision tables as tables? That would be cool…
You should take a look at Jetbrains MPS, I believe that’s pretty much what you’re looking for. I think you’re underestimating the value of text files, though: It’s not just plain old stubbornness that makes people continue to use them, it’s the vast ecosystem of tools surrounding this concept.
DrScheme (now DrRacket) allows XML and images to be embedded in the REPL.
http://plt-scheme.org/screenshots/graphical-syntax.jpg
Excellent post – one of the best I’ve ever seen. I have been developing software for nearly 40 years (I do remember the early articles on literate programming, and that at the time I believed it had a future). But I agree with you – many of the things you mentioned I have come to the same conclusion, but you have described it so much better than I can, plus added a few things I had not thought about.
One approach I have been thinking is that during the development phase the source code can be kept within a specially designed webdav server, and stored as “code” blocks in wiki pages. In the wiki page but outside the code block one can add anything else they like, including graphics, tables, etc. The webdav server would strip all the non-code stuff if the request comes from a compiler, and keep it all if the request is from a regular browser.
I must say this is not entirely my idea – it is inspired by genius Ward Cunningham’s FIT http://fit.c2.com/
He uses wiki pages to hold test cases – the tests are in tables, anything else is documentation. The FIT framework knows how to extract the test cases from the tables.
Also, there are many other benefits to using webdav, such as the built-in versioning.
About the only problem might be extra delay for using an http request and a bit extra processing to extract the source code – which might affect worst of all a build, but these days this should be of little consequence when compared to all the other benefits.
Maybe some young person with a bit of enthusiasm and spare time might like to experiment (I am afraid my age is catching up with me).
Thanks so much – cheers – Luciano
Maybe what you’re looking for is not a new programming language, but an IDE that integrates documentation with (textual) compilable code. Markup in the code could reference JPEGs, web pages, whatever artifacts you need, and vice versa. I’d agree that graphical programming ala the Rational toolset hasn’t really taken off. And text (and text processors) are ubiquitous, with good reason: written text is a powerful, easily-understood communication medium. CAD applications work well with physical things like HVAC systems and electronic circuits that have their own well-defined graphical symbolic vocabularies. Maybe business processes could be modeled and programmed effectively with CAD-like tools. But try to describe handling of laboratory data, or a medical record exclusively through CAD diagrams. Sometimes (lots of times) written text–narrative, code–is the simplest, most concise means of expression.
Maybe the mechanical/architecture/ic layout guys get benefit from the nice graphical CAD tools but unfortunately, the electrical design guys are stuck using wonderful “programming” languages like Verilog/SystemVerilog/VHDL which look an awful lot like C/C++/Ada.
Those RTL guys just use flowing text and not graphics … yes there are pretty graphics tools but they see about as much use as tools like STATEMENT and other graphical software tools.
The graphical tools are all “too restrictive” and do not all the designer to build efficient hardware. I saw we all go back to the hardwired plug boards for programming and tube based designs for hardware. Those were the days
We are all just a bunch of Luddites.
I agree with Stefan, MPS might be in the direction you want to go. At least, the programming model is quite different from anything I’ve seen before (well, that’s not _that_ much, but hey…). You basically edit the AST in any form you like, tables being one of them. It’s a tool to create DSLs and uses representations you defined for the different Elements. Hence you might even be able to use graphics for programming.
Last time I looked, they seemed to have a mission. As one of the last software development tool vendors that still manage to make money from their tools, they might even be able to pull it off.
Dear greg, you might like to see: “How to write Computer programs”
at http://www.civilized.com/programming.html
- Your point that plain documentation is unchecked is true, but
disciplined behavior can obviate that. And I find that reading
undocumented code is very painful – even my own – but of course,
my own is not undocumented. (Also see:
http://www.civilized.com/files/msgfuncs.txt)
all the languages are taking ideas from lisp more and more often….
Greg, I think you’re being a bit facetious here. Just a few posts after this one, you complain about (http://software-carpentry.org/2010/11/ratios-and-rework/), you complain about Powerpoint and how it doesn’t play nicely with version control. Would you like your code to *not* play nicely with version control, too?
I agree with you that there should be a better way to view inline documentation — the IDEs cannot implement this soon enough. I would still prefer to write it in Markdown, but there should also be a WYSIWYG editing option. 20-odd years ago, WordPerfect had a mode that revealed the formatting codes (the equivalent of HTML tags) in a document. This type of feature disappeared when GUIs took over, but widespread use of wikis makes it attractive again. There are already at least a few “high-end” tools — Confluence for example — that offer both WYSIWYG and “source” editing modes, and I hope this makes the jump (back) to text editors as well.
Why do you think the two are mutually exclusive? PPT and MP4 and every other format ever invented could play nicely with version control—someone just has to build a diff/merge engine. Just as an example, there are several commercial diff/merge tools for Excel spreadsheets, and at least one high-end constraint-based CAD system comes with one built in.