Python, readline and insanity (mine)

A bit of a technical post here. For the last week or two I've been writing some financial modelling software in Python, to see whether it's really possible to make serious money from a rather clever scheme that a friend of mine came up with. When I came to put the command line interface (CLI) together, I wanted something with all of the usual Unix CLI stuff - recall, editing, completion, help. The standard way to do this for C/C++ code is using the GNU readline library, which is what the Linux shell uses for example.

I recently got round to doing this for another piece of software, written in C++. There, though, I couldn't use readline. Readline is licensed under GPL, which means, essentially, that if you use it in your code, everything you write legally enters the public domain. Luckily there is a non-GPL near-equivalent, called editline. This provides all of the useful functionality of readline - which actually is about 5%, like most software 95% of what it can do is unknown and probably incomprehensible to 95% of the people who use it (just like Word or Excel).

I had no such constraint with my Python code. Python provides a module for interfacing to readline which makes the basics - command recall and editing - incredibly simple to use. Simply importing the module changes the behavior of the console input function raw_input to provide all this. Nothing could be simpler.

But I wanted more. Specifically, I wanted tab completion and also Cisco-like help, where typing '?' prompts with what to enter next. In C, completion is done using a callback, and you can define different completions for different functions - in particular, you can have one for tab and another for '?'. But the Python mapping provides only a single callback. More on that later.

The implementation looked simple. One function call (set_completer) to register the callback function, another (parse_and_bind) to tell readline how to handle the special characters. I found a good example and pretty much copied it. That's where the insanity comes in. The example, cut and pasted into a file, worked perfectly. My code didn't. Hitting tab did nothing at all. Putting diagnostic code into the completion handler produced noting. Even deliberate errors - like using unassigned variables - did nothing. My code simply wasn't being called. There followed a couple of increasingly desperate hours of cutting and pasting, copying code from the working example to the non-working code and vice versa... all to no avail. The example worked, my code didn't - even when they were identical!

I'd set aside an hour or so to get all this stuff done, and by now I was into the third hour and still getting nowhere. I was just a little frustrated. Finally, by application of the Sherlock Holmes Principle ("once you have eliminated the impossible, whatever remains, no matter how improbable, must be the truth") I realised what was happening.

The Python interpreter makes heavy use of exceptions, both for intended cases (e.g. key not in a dictionary) and for programing errors (e.g. referencing an undefined variable). Normally, an uncaught exception - from an error - ripples right out to the outermost function, where the interpreter prints a stack dump. You know not only that something bad happened, but where. Things get different when C code gets in the way. Python passes the exception down into the C code, but what happens to it there depends entirely on the C code. And in the case of the readline library, the C code just swallows the exception and says nothing. So if there's a bug in the completion handler, it just fails silently.

Once I realised this, the solution was obvious - put a catch-all handler in the outermost completion handler function. That tells what the error is, though it doesn't give a stack dump. Once I did that, I could see the errors and quickly had things running.

There was another problem though, which I alluded to earlier. Writing in C, you can have a distinct callback for tab and '?'. In Python, there's no way to do this, and no other way to find out which character was typed since it doesn't get put anywhere. There's simply no way to know which it was. I did find a patch to the Python readline module which made this possible, but I don't want to deal with a special version of the code. So my compromise is to treat them exactly the same. If either occurs as the first character of a field, it's treated as '?', and generates a helpful message telling you what's expected. Once you've typed anything else, what you get is completion.

So, the moral of this is, always put a catch-all exception handler in the outermost Python callback function, and life will be good.