Lately I’ve been doing most of my testing with doctest, primarily using stand-alone text files. I generally like it (otherwise I wouldn’t be using it), but it does make me frustrated with doctest sometimes. On my wishlist (roughly in order):
-
I wish output was always displayed, even when there’s an exception. I see no reason for the current behavior. Really exceptions could be treated like any other output (if ELLIPSIS was on by default).
-
I wish you could turn on options like ELLIPSIS from within a doctest, for all expressions. (# doctest: +ELLIPSIS on every line is beyond ugly.)
-
<BLANKLINE> is terribly ugly.
-
There’s no way of saying "I don’t care what this prints". You can’t do:
>>> some_function() ...
because the … is treated like a continuation.
-
Plugging in an alternate output checker is kind of tedious, and can’t be done from within a doctest (without horrible hacks).
-
I’d like to be able to easily jump into an interactive state from doctest. Maybe pdb can do this, but I’ve never figured that out exactly.
-
Getting nose to run .txt files as doctests is really hard, involving a combination of options I always forget.
-
There’s no way to abort the doctest. Sometimes I’d like to run some environment checks early on, and be able to stop the test if they fail.
-
I wish it was easier to apply to non-Python code. (I’ve adapted it via subclassing for Logo but I wouldn’t do that often.)
-
I wish I could copy and paste from doctests to consoles. But I don’t see any solution to this problem.
-
The integration with unittest is pretty hacky. Not that I’ve used unittest in years. But some other test frameworks build off this integration.
-
python -m doctest sometest.txt doesn’t do what it should do. Instead it runs doctest’s self-tests.
Tarek Ziadé | 31-Jul-08 at 11:42 pm | Permalink
“Getting nose to run .txt files as doctests is really hard, involving a combination of options I always forget.”
Yes that is a problem, the nose plugin is not really handy. We ended up faking regular test cases to be able to run our .txt doctests easily with Nose.
see http://www.gawel.org/weblog/2008/07/nose-doctest-plugin-sucks for an example of such a script.
I think Nose could be enhanced there.
“There’s no way to abort the doctest. Sometimes I’d like to run some environment checks early on, and be able to stop the test if they fail.”
You could do a simple assert on your test, with the
REPORT_ONLY_FIRST_FAILUREflag on. The assert can be wrapped in a global function you provide in the test environment.Paddy3118 | 01-Aug-08 at 2:53 am | Permalink
On dropping into the interpreter, Duncan Booth set me strait here.
Gavin Panella | 01-Aug-08 at 6:05 am | Permalink
You can drop into
pdbwith the usualimport pdb; pdb.set_trace(), but, in my experience, you land in the middle ofpdb’s code. Justnextto get back to the doctest scope.ogrisel | 01-Aug-08 at 6:09 am | Permalink
The zope.testing [1] testrunner launched with the -D option drops pdb as postmortem shell when a test fails. This test runner can be used easily for non-zope related project. This is really a huge time saving feature when doing TDD.
[1] https://launchpad.net/zope.testing
Ian Bicking | 01-Aug-08 at 9:02 am | Permalink
Some notes:
Stuff like
pdb.set_trace()is good, but it’s better when you can do it when the test fails (what zope.testing and nose can do with options). Interaction isn’t part of the test, it’s part of the testing process.Something I forgot to mention:
REPORT_UDIFFalso has this problem. You have to put the option into the test to see the diff’d output, but it’s really part of the testing process, something you apply when you can’t quickly see the differences in output.REPORT_ONLY_FIRSTdoesn’t work: it still runs all the other examples, it just doesn’t report them. This is often fine, but sometimes I really want to abort the test. A common example would be a database-oriented set of tests, when I can’t open the database connection. Sometimes when testing destructive file operations it’s actually dangerous to continue the test if you can’t confirm that you are in a safe scratch area.slinkp | 01-Aug-08 at 10:06 am | Permalink
Regarding pdb, a tip I learned from Rob Miller is that you want to put your set_trace() on the same line as the code you want to step into:
>>> import pdb; pdb.set_trace(); foo()
And once you step out of foo(), you can still step through the doctest, but stupidly you can’t see what line you’re on. This sucks.
Jason Pellerin | 01-Aug-08 at 10:59 am | Permalink
“Getting nose to run .txt files as doctests is really hard, involving a combination of options I always forget.”
Me too, which is why I put them in setup.cfg and/or ~/.noserc:
“see http://www.gawel.org/weblog/2008/07/nose-doctest-plugin-sucks … I think Nose could be enhanced there.”
Unfortunately, I don’t speak French, so I’m only guessing that the post there is complaining about the lack of support in the doctest plugin for fixtures. If that’s the case, some support is scheduled for the next nose release:
http://code.google.com/p/python-nose/issues/detail?id=60
You can find the current state of fixture support in the ticket-93 branch in nose’s svn. If you find it still too sucky, patches are always welcome.
Stefan Eletzhofer | 01-Aug-08 at 2:37 pm | Permalink
” … I wish I could copy and paste from doctests to consoles. But I don’t see any solution to this problem. … “
I use IPython. It solves quite some problems:
allows for pasting doctests and evaluating them (IPython.dtutils.idoctest), optionally
raising a exception on the first failure
doing a post-mortem pdb automatically
executing the doctest in a separate namespace
switching to a “normal” prompt (ipython normally has a fancy, colored, history-enabled prompt), so you can paste code from the interpreter to a doctest. One might consider this not a new feature, because the standard interpreter has this too :P
Btw, you can invoke a IPython-enabled pdb by using http://pypi.python.org/pypi/ipdb and:
from ipdb import settrace; settrace()
I like IPython :)
Stefan.
Noah Gift | 04-Aug-08 at 6:28 am | Permalink
I will second Stefan’s comments about IPython, it has changed the way I write Python code, and as such doctest is even more important. Fortunately IPython has a form of support for doctest. I think there are some improvements that could be made, but I find it very useful to go into doctest mode in IPython to try something out, then when I am done, paste that code into a test.
Tarek Ziadé | 09-Aug-08 at 12:22 am | Permalink
@Jason Pellerin
Thanks for the info, I’ll give a shot