My first Rest API

For several months I've been working on an application that I always knew would sooner or later need a Rest API added to it for configuration and monitoring. These days, if you don't have a Rest API, you're nobody - even Cisco has added one to IOS. This would be the first one I've created myself.

My goal was to make this as simple and as little work as possible. My only previous experience of Rest APIs was the exact opposite. The designers had gone out of their way to make the Rest API as complex and burdensome to implement as possible, spending whole days in minute reviews of parameter names and metadata like XSD files. They had also built an exceedingly complex implementation - a perfect illustration of the French expression une usine à gaz. The bits that needed to have high performance were in PHP, while the low performance string handling was done in C. Honest. Python was in the mix somewhere too, and you could probably find Cobol and APL if you looked hard. It was a big mess, and the exact opposite of what I wanted to achieve.

My main goal was to do as little work as possible, both in building the initial infrastructure and even more so when making additions later. I'm a one-man band on this project, 15,000 lines of code in the last six months, and the less work it takes to do a relatively peripheral thing like this, the better. That said, I'm always willing to spend a little longer getting the infrastructure right if it makes less work later on, and especially if it makes for less error-prone repitition.

I wanted to make it as "truly Restful" as possible, respecting the Rest orthodoxy. Though this turns out to be harder than you'd think. Most writings on the subject are distinctly obscurantist, leaving me with the same feeling as when I try to understand writings on philosophy or (worst of all) sociology - I know what all the words mean and I can pretty much figure out the sentences, but I have no clue what they are actually trying to say. A principle which is held in especially high esteem is called HATEOAS. People write pages and pages about how bad it is not to follow this principle, but nowhere have I found an example which illustrates what it actually means. I think it means that you should have lots of hyperlinks to associated resources, which is a good idea in human-oriented web stuff too. So that's what I did, although honestly it might mean something completely different.

My underlying code is all written in C++. The configuration interface is exposed through a singleton policy_manager class, which has functions like add_interface or show_acl that access the underlying C++ objects. I'm a big fan of using Python to do anything which isn't performance critical, so the first thing to do was expose this as a set of Python functions. Boost::python to the rescue for this part - it took only an hour or so to expose these functions in Python.

One thing I really did not want to do, was to have to repeat each function declaration over and over for different parts of the interface. I already had the usual .h/.cpp files for the C++ code. It was a 15 minute task with emacs to transform the function declarations in the .h file into some macros that captured the essentials of the functions, i.e. the names and details of the parameters, in a way that could expanded differently as required in the different places.

The C pre-processor (CPP) is very limiting in what it can do. To me the nec plus ultra of integrated macro facilities has always been the DEC assemblers. They didn't do much more than CPP, but the little there was made a huge difference to the power and flexibility. Of course there's always M4 - I used it for a big project once in the past and it is amazingly powerful, but I'd prefer to do without the extra build complication, not to mention remembering how to use it.

I came up with what amounts to an Application Specific Language to describe my functions. It's not especially pretty but it gets the job done. Oh, for a "shift" function (like the shell) to deal elegantly with variadic parameters.

All that done (it actually didn't take very long), I wanted an RPC so that my Rest server didn't have to be in the same process as the operational stuff. Pyro was the answer to that. It's a truly amazing little package, that lets you export a Python object with close to zero effort. My RPC server is about a dozen lines of code. It creates the Python object corresponding to my singleton policy_manager, and exports it. Problem solved, with amazingly little work.

The next step was to select a framework for the Rest server itself. I looked at Django, but it's really designed to do a lot more than just serving web pages, and correspondingly rich and complex. I settled on Flask, which has a very intuitive and simple way of relating URLs to the code that serves them.

To minimize the amount of work needed, I made every class work the same way. For the POST method, there is an add_... function, for GET a show_... function and so on. So the heart of the server is a table which maps Rest prefixes to the family name of the function. As a trivial example, the prefix 'interfaces/' maps to "interface", so a GET to 'interfaces/' results in a call to show_interface. There's some generic code to do things like extracting names from URLs (e.g. 'interfaces/eth0/hosts/1.2.3.4') then depending on the verb to construct the corresponding function call.

One tricky point was the construction of links in the Rest output. The underlying C++ code knows nothing about Rest, and certainly not about the specific URL structure in use, and I want to keep it that way - so it can't generate explicit links. The solution is to pass a class name as well as the leaf instance name - e.g. {'class':'interface', 'name':'eth0'} - as the value for a link to another object. In the C++ code these attributes are pointers, so the generic attribute output code understands that pointers should be passed in this form. Then the Rest server code has a post-scanner that takes the output from the show_... functions, and using its class-to-URL mapping table, turns these into full-formed URLs.

One nice thing is that the Python code in the Rest server has no idea what is and isn't available for each class. If a class doesn't support POST, for example, then the server will generate a call to add_<class> anyway. Pyro will eventually determine that there is no such function, returning a corresponding error, which the Rest server turns into a 405 (method not permitted) error.

I made a decision early on to support only Json. This seems to be pretty normal now. XML generates a lot more work around metadata, for no added functional value. It wouldn't be hard to also generate XML, but it's hard to see why it would be useful.

In the end it took about three days of work, spread out over the Christmas and New Year break, to get a fully functioning Rest interface. There has been quite a bit more work since in the backend, as I've realised how information should be presented. The Rest server is about 250 lines of Python. Each new class adds exactly one more line, to the mapping table. And there's no PHP!

Python 3 - what were they thinking?

I'm a huge fan of Python for all kinds of programming, just about anything that doesn't need high performance or huge data structures. At my new company we are using it for all kinds of things. It's because I like Python so much that I'm so mad at the core team for forcing the Python 3 do-I-or-don't-I dilemma on the world (and more specifically, on me). I was forced this week to make the company-wide decision as to which dialect to use. There's no really good answer, though in the end I decided on Python 2 rather than Python 3.

What makes me angry is that the need to choose was forced upon the world by the sheer arrogance of the Python team. There's absolutely nothing in Python 3 that couldn't have been done in a backward-compatible way, allowing a gradual migration. Instead, they chose to make numerous changes which make it impossible to write code which will compile either as V2 or V3 of the language.

If you're just writing your own code, you can of course choose either version. But the big advantage of Python is the vast number of libraries for doing just about anything you can think of. And if you want to use those, they have to be available in your chosen dialect. According to a blog I read, less than a quarter of all libraries have been converted, including some major ones such as Django, which presents databases as web pages. Or if they have been converted, the paint is still very wet on the conversion and there are likely to be performance issues or just plain bugs. Another blog started out by promoting the idea of using V3, but ended up by saying that if it's for serious production, you're better off staying with V2. As it happens we don't need Djano, matplotlib, or numpy (three of the biggest libraries), but you never know what's round the corner, or what nifty library you'd like to use but can't because its developers haven't made the effort to do the porting - and after all, why should they, given that much of this code is written by volunteers, for fun.

Yes, there is one fundamental change which it would be hard to do in a compatible way, which has to do with the representation of strings. But for most uses of the language, this doesn't really matter. Only if you deal with transformations between raw bytes and strings do you really need to be exposed to this. There would certainly be a way, using the "future" concept, to allow the minority of code that needs to know about this to deal with it in a compatible way.

All of the other changes are just change for change's sake. Yes, maybe the new form is better in some ivory-tower sense, but not enough to justify the millions of hours of work that are being expended on the migration. And all of them could have been done by introducing a new form, then deprecating the old form over a period of years.

So this is purely a "because we can" exercise by the Python team, to show us all how clever and how powerful they are. It's enough to make you go out and learn Ruby.

Python and Tkinter: wonderful

Having time on my hands at the moment, and no work commitments - that's another story though - I decided to start taking a new look at the robotics stuff I was playing with a year or so ago.

I'd written nearly all of the code to make a six-legged robot - a hexapod - walk with various different gaits and postures - the so-called inverse kinematics. It was in straight C, since I intended it run on a little embedded CPU which had no support for C++, nor floating point for that matter. And I'd built a development environment, including visualisation for the leg movements, using Visual Studio.

Things have moved on since then, though. For one thing I've pretty much switched to Linux for my computing environment. For another, the Roboard has really become the obvious onboard computer - it is now available with Linux, and it offers a full-function x86 including floating point, in a tiny size that will fit in my fairly small hexapod. And since it supports full GCC, I can write the code in C++. The C code is just so cluttered - I can't for the life of me imagine why anyone would prefer to code in C. It's full of irrelevant details that make it hard to read and even harder to get it to work. So, a rewrite is called for.

The only problem, is the GUI that I'd painfully created using the Visual Studio tools. Painful because there are 6 legs, and each has numerous parameters and state variables. I'd created the dialog box from hell. Every tiny change meant nudging numerous components around to get it to look right. What a pain. But it was done, for now anyway.

That was when I thought about Tkinter, which I've never used before. I've become a huge fan of Python in the last couple of years, using it for anything where performance is not a big deal. I also wrote a very powerful Python-based scripting system for my now-defunct employer, using Boost Python. So using Python and Tkinter for the GUI was kind of an obvious thing to do.

Somewhere in the mists of history I acquired Python and Tkinter Programming, which I think is the definitive book on the topic. I skimmed that, and with frequent help from Google - especially this site - started putting my new GUI together.

What a pleasure! Tkinter automatically takes care of making a reasonable layout, given some general guidance through the pack and grid functions. You no longer have to think about the minutiae of positioning, or spend ages getting boxes to line up with each other. I just couldn't help putting together a bit of infrastructure for collections of config variables, so they are now super-easy - just a list of names and default values and Python and Tkinter take care of everything.

In total it has probably taken me about 6 hours to get everything together - but that included learning Tkinter from scratch and writing quite a bit of infrastructure. And now I have everything I need to control my inverse kinematics, and have an animated visualisation of what it's doing.

I'll never do GUIs any other way now. Tkinter is wonderful!

Python, readline and insanity (mine)

A bit of a technical post here. For the last week or two I've been writing some financial modelling software in Python, to see whether it's really possible to make serious money from a rather clever scheme that a friend of mine came up with. When I came to put the command line interface (CLI) together, I wanted something with all of the usual Unix CLI stuff - recall, editing, completion, help. The standard way to do this for C/C++ code is using the GNU readline library, which is what the Linux shell uses for example.

I recently got round to doing this for another piece of software, written in C++. There, though, I couldn't use readline. Readline is licensed under GPL, which means, essentially, that if you use it in your code, everything you write legally enters the public domain. Luckily there is a non-GPL near-equivalent, called editline. This provides all of the useful functionality of readline - which actually is about 5%, like most software 95% of what it can do is unknown and probably incomprehensible to 95% of the people who use it (just like Word or Excel).

I had no such constraint with my Python code. Python provides a module for interfacing to readline which makes the basics - command recall and editing - incredibly simple to use. Simply importing the module changes the behavior of the console input function raw_input to provide all this. Nothing could be simpler.

But I wanted more. Specifically, I wanted tab completion and also Cisco-like help, where typing '?' prompts with what to enter next. In C, completion is done using a callback, and you can define different completions for different functions - in particular, you can have one for tab and another for '?'. But the Python mapping provides only a single callback. More on that later.

The implementation looked simple. One function call (set_completer) to register the callback function, another (parse_and_bind) to tell readline how to handle the special characters. I found a good example and pretty much copied it. That's where the insanity comes in. The example, cut and pasted into a file, worked perfectly. My code didn't. Hitting tab did nothing at all. Putting diagnostic code into the completion handler produced noting. Even deliberate errors - like using unassigned variables - did nothing. My code simply wasn't being called. There followed a couple of increasingly desperate hours of cutting and pasting, copying code from the working example to the non-working code and vice versa... all to no avail. The example worked, my code didn't - even when they were identical!

I'd set aside an hour or so to get all this stuff done, and by now I was into the third hour and still getting nowhere. I was just a little frustrated. Finally, by application of the Sherlock Holmes Principle ("once you have eliminated the impossible, whatever remains, no matter how improbable, must be the truth") I realised what was happening.

The Python interpreter makes heavy use of exceptions, both for intended cases (e.g. key not in a dictionary) and for programing errors (e.g. referencing an undefined variable). Normally, an uncaught exception - from an error - ripples right out to the outermost function, where the interpreter prints a stack dump. You know not only that something bad happened, but where. Things get different when C code gets in the way. Python passes the exception down into the C code, but what happens to it there depends entirely on the C code. And in the case of the readline library, the C code just swallows the exception and says nothing. So if there's a bug in the completion handler, it just fails silently.

Once I realised this, the solution was obvious - put a catch-all handler in the outermost completion handler function. That tells what the error is, though it doesn't give a stack dump. Once I did that, I could see the errors and quickly had things running.

There was another problem though, which I alluded to earlier. Writing in C, you can have a distinct callback for tab and '?'. In Python, there's no way to do this, and no other way to find out which character was typed since it doesn't get put anywhere. There's simply no way to know which it was. I did find a patch to the Python readline module which made this possible, but I don't want to deal with a special version of the code. So my compromise is to treat them exactly the same. If either occurs as the first character of a field, it's treated as '?', and generates a helpful message telling you what's expected. Once you've typed anything else, what you get is completion.

So, the moral of this is, always put a catch-all exception handler in the outermost Python callback function, and life will be good.