Ian Bicking: a blog

Lately there’s been some interest in build processes among various people — Vellum was announced a while back, Ben has been looking for a tool and looking at Fabric, and Kevin announced Paver. At the same time zc.buildout is starting to gain some users outside of the Zope world, and I noticed Minitage as an abstraction on top of zc.buildout.

A while ago I started working on a build project for Open Plans called fassembler. I think the result has been fairly successful and maintainable, and I thought I’d share some of my own reflections on that tool.

$ build-something target-directory/
(much time passes)
Error: cannot write /home/ianb/builds/20080426/target-directory/products/AuxInput/auxinput/config/configuration.xml

People have asked me a few times about evalexception and pdb — they’d like to be able to use something like pdb through the browser, stepping through code.

The technique I used for tracebacks wouldn’t really work for pdb. For a traceback I saved all the information from the frames — mostly just the local variables — and then let the user interact with that through the browser. But with pdb you pause the application part way through waiting for user input, and the routine only completes much later.

While writing WaitForIt I played around with techniques to deal with very slow WSGI applications. Not that hard, really — you launch every request in a new thread, and you manage those requests in an application of its own. So I started thinking about pdb again, and it started seeming feasible. Whenever the app reads from stdin it goes into an interactive mode, showing you what comes out on stdout and letting you add input to stdin. It’s nothing specific to pdb really.

So, with a bit of hacking, I added it into WebError (which is an extraction of the exception handling in Paste). To give the demo a try, do:

hg clone http://knowledgetap.com/hg/weberror/
cd weberror
python setup.py develop
# You need Paste trunk:
easy_install Paste==dev
python weberror/pdbcapture.py

What you’ll see is not polished, it’s just working, but since I mostly did it to see if I could do it, that’s good enough for me.

My old do-it-yourself framework tutorial was getting a bit long in the tooth, so I rewrote it to use WebOb. Now: the new do-it-yourself framework.

So I promised some more technical discussion of App Engine than my last two posts. Here it is:

Google App Engine uses a somewhat CGI-like model. That is, a script is run, and it uses stdin/stdout/environ to handle the requests. To avoid the overhead of CGI a process can be reused by defining __main__.main(). But while a process can be reused, it might not be, and of course it might get run on an entirely separate server. So in many ways it’s like the CGI model, with a small optimization so that, particularly under load, your requests can run with less latency.

This part is all well and good. I’ve already come to terms with servers going up and down without warning. But the environment itself has a number of other restrictions. It seems that App Engine is providing security in the language itself. The interpreter has been modified so that code is sandboxed, with no ability to write to the disk, open sockets, import C extensions, and see quite a few things in its environment. It’s these things that are a bit harder to come to terms with.

While they claim it supports any Python framework, these restrictions don’t actually make it easy. So for the last few days quite a few of us have been hacking various things to get stuff working.

The first thing people noticed is that Mako and Genshi didn’t work, because they use the ast (via the parser module) to handle the templating, and that module has been restricted. Apparently arbitrary bytecode is not safe in this environment, and so anything that can produce bytecode is considered dangerous. From what I understand Philip Jenvy has been working on Mako and the trunk is currently working. He’d already been doing work to get Mako working on Jython, which had similar issues. Genshi is also in progress and fairly close to working, though with some missing features. Genshi has the harder task as Mako was primarily reading the ast, while Genshi was writing it.

The first thing I noticed is that Setuptools doesn’t work. I’m flattered that one of the only 3 libraries included with App Engine is WebOb, but of course I am more enamored of a rich set of reusable libraries. Setuptools didn’t work because several modules and functions have been removed — this like os.open, os.uname, imp.acquire_lock, etc. Some of these are kind of reasonable, while others are not. The removal of many functions from imp doesn’t really make sense, for instance (I think the motivation was the difficulty of auditing the implementation of those functions, not that the functionality itself is dangerous). And while some functions can’t be used in the environment, the fact you can’t import those functions is more problematic. For instance, The Setuptools’ pkg_resources module has support to unzip eggs when they are imported. App Engine doesn’t support importing from zip files at all, and you certainly can’t unzip to a temporary location. But withoutthe necessary modules and objects pkg_resources won’t even import.

To work around this I started a new project: appengine-monkey, which adds several monkeypatches and replacement dummy modules to the environment to simulate a more typical environment. It’s just a small list so far (mostly in this module), but I expect as people experiment with other libraries the list will increase. For example, I would welcome implementations of things like httplib on top of urlfetch in this library. (Implementing httplib and stubbing out parts of socket would probably make urllib run.)

But the good news is that Pylons is pretty much working on App Engine, as is Setuptools and you can manage your libraries using virtualenv.

The instructions are all located in the appengine-monkey Pylons wiki page. Please leave comments if you have improvements or problems with that process. I also welcome contributors and developers to the project itself — this is a project for expediting App Engine development, it is not a project I care to champion or control. Or support to any large degree.

One ticket which is rather important is the apparent maximum number of files and blobs: 1000. Libraries involve lots of files, and the base Pylons install is only barely under this limit. Now I just wish I could use lxml, but that’s probably going to be a long time coming.

I like this phrasing of the debate about App Engine’s role, from Doug Cutting: Cloud: commodity or proprietary? (via). (Well, I like it more than the sharecropping phrasing referenced in my last post.) I guess I’m excited about this because like Doug I do want a "cloud" of sorts, and this is a move towards that in a way that makes sense to me. Maybe to state my motivations more clearly: I hate computers. I really hate them a lot. I dream of some world of Platonic ideals where software just exists, and existence that state it works. App Engine feels like a strong move in the direction of computers-not-mattering. What does App Engine run on? I don’t care! Where is the server located? I don’t care! What is BigTable? I am comfortable thinking of it only in its abstract sense, an API that works, and I don’t know how, nor do I need to know how. I don’t need to know these things if they just work. Always. Totally reliably. I’m not shy about digging into code. I tend to be light on my reading of documentation, because I’d much rather open up source code and poke around. But when something can work so reliably that I can treat it as completely opaque then it’s a blessing, because I can start to forget about it and think about bigger goals.

There was a time when people were concerned about Big Endian vs. Little Endian in computers. You had to think about this sort of little detail when programming. People formed actual opinions on which way was best. To think! Similarly XML has removed a large number of fairly pointless format decisions people might make. There is progress. Commodity hosting (reliable, consistent hosting, better than what we have now) feels like similar progress.

Unlike Doug I’m optimistic that App Engine is a move in the direction of a commodity cloud. The APIs seem to lack the stench of proprietary APIs. They are based on Google services, but they reasonably abstract and reasonably minimal. This does not seem like some kind of "play" (and the developers’ seem to be reassuring about their intentions). There’s a tendency to be cynical about any company’s work, that it has underlying intentions that are at odds with any competitor (present or future), that anything good is just a loss leader meant to hook you in so they can squeeze you later. Some companies deserve such cynicism. I don’t know that this company, or this team, deserves that.

Mind you, I don’t say this from a Best-Tool-For-The-Job perspective. I believe in the moral foundations of Free Software, not just the technical advantages of its development process. But I’m a Utilitarian, and it doesn’t make me uncomfortable that not everything is Free if I think it’s a step forward for overall freedom. I think App Engine has the potential to be a very powerful tool for enabling people to create and use web applications. If it was great, but still a dead end, then maybe that wouldn’t be good enough. But I don’t think this is a dead end.

Update: Indeed people are reimplementing the interface: see the appdrop.com announcement

This is about Google App Engine which probably everyone has read about already.

I’m quite excited about it. Hosting has been the bane of the Python web world for a long time. This provides a very compelling hosting situation for Python applications.

I’m not as interested in this from a competitive perspective as I am from a simple this-is-awesome perspective. Regardless of how this positions Python relative to other languages, this is something Python needs. But even looking beyond that, I think this is something the open source world needs. Open source web development is in a funny place. There’s a lot of reasons why web programming is a good domain for open source. The barrier to entry for web development is extremely low. Developers have choice in their tools, as browsers don’t really care what software you use so long as you serve up HTML. This leads to experimentation and excitement and the kind of self-direction that is very motivating to developers. It leads to the kind of personal excitement that underlies most open source development.

Despite this, open source web application development doesn’t seem sustainable. There’s some applications, sure. WordPress, Trac, MediaWiki, MoinMoin. But most wiki software doesn’t have a vibrant community. Many a bug tracker has fallen by the wayside. Blog software projects have a horrible time building a viable community. Other website software hardly gets anywhere at all. A lot of the development that might appear to be application development really is more like a framework when you look closely (e.g., Plone, Drupal).

I think deployment concerns are a huge part of this. And, given its better deployment story, it’s no surprise PHP is the basis of most viable open source web applications. Being interested in a project requires that you be able to use the product (and usually use it casually, as that’s the point of entry for many developers). Right now most people can’t use open source web applications.

But people can use hosted applications, and that’s where all the effort has gone in the past few years. I am comfortable saying that Trac is a better issue tracker than Google Code’s issue tracker. But I’d probably recommend Google Code to someone starting a new project, because it’s so much less work. Similarly I’d try to dissuade most people from installing their own blog software. I still don’t know what to tell people about a CMS.

Many people are excited about how far up you might be able to scale something based on App Engine, but (like Dave) I’m excited about how far it could be scaled down. For the majority of sites the free quota will be more than enough. But that alone isn’t the point: there’s lots of free services people can use. The difference here is that the free services can be modified and controlled by the anyone who signs up and installs an application.

From the perspective of open source it’s a bit awkward that the platform itself is proprietary. Questions about sharecropping are a valid concern, but I’m optimistic about the ultimate outcome. The SDK is under a permissive open source license, and the APIs are all reasonable enough that they could be reimplemented with open source backends (maybe without the same scalability, but that’s not the aspect I care about anyway). Perhaps the BigTable APIs will serve as the basis for future storage APIs.

But even if other people make compatible implementations of these APIs, would it matter? If Google offers free hosting, is someone else really going to be able to provide a better hosting option? Or would these other implementations just be strawmen, a way to show that It Could Be Done? If the libraries are just written to prove a point, I can’t see them gaining much traction. But I think these could be viable as there are other constraints to the App Engine environment that people may want to escape at some point in their application development.

As to the details of App Engine? Can you run Pylons or Paste on it? Well, that’s a topic for another post.

Update: I wrote up some more thoughts

I just saw this json-rpc recipe go by as a popular link on del.icio.us. It’s yet-another-*Server based recipe (BaseHTTPServer, XMLRPCServer, etc). I don’t know why people keep writing these. WSGI is in all ways easier, clearer, and more useful.

So I figured I’d give it a go myself, using WebOb. Then I figured it might make a good document, and I annotated it and turned it into a tutorial. It’s also an example of using WebOb as a client library.

I’ve added several tutorials in the past months, which I thought I should also point out. The wiki example is fairly pedestrian, but shows how to do typical application-style development with WebOb. A more interesting exampe is the comment middleware example, which shows how much easier it can be to write middleware using WebOb than traditional WSGI middleware.

I think WebOb’s particular strong points are with middleware (where it makes middleware vastly easier to write) and web services of various kinds (RESTful or not).

So I gave a presentation at PyCon about HTML, which I ended up turning into an XML-sucks HTML-rocks talk. Well that’s a trivialization, but I have the privilege of trivializing my arguments all I want.

Somewhat to my surprise this got me a heckler (of sorts). I think it came up when I was making my <em> lies and <i> is truth argument. That is, presentation and intention are the same. There are those people who feel they can separate the two, creating semantic markup that represents their intent, but they are so few that the reader can never trust that the distinction is intentional, and so <i> and <em> must be treated as equivalent.

Someone then yelled out something like "what about blind people?" The argument being that screen readers would like to distinguish between the two, as not all things we render as italic would be read with emphasis.

It’s not surprising to me that the first time I’ve gotten an actively negative reaction to a talk it was about accessibility. When having technical discussions it’s hard to get that heated up. Is Python or Ruby better? We can talk shit on the web, where all emotions get mixed up and weirded, but in person these discussions tend to be quite calm and reasonable.

Discussions about accessibility, however, have strong moral undertones. This isn’t just What Tool Is Right For The Job. There is a kind of moral certainty to the argument that we should be making a world that is accessible to all people.

I fear this moral certainty has led people self-righteously down unwise paths. They believe — with of course some justification — that the world must be made right. And so many boil-the-ocean proposals are made, and even become codified by standards, but markup standards are useless unless embodied in actual content, and this is where accessibility falls down.

There are two posts that together have greatly eroded my trust in accessibility advocates, so that I feel like I am left adrift, unwilling to jump through the hoops accessibility advocates put up as I strongly suspect they are pointless.

The first post is about the longdesc attribute, an obscure attribute intended to tell the story of a picture. Where alt is typically used as a placeholder for the image, and a short description, longdesc can point to a document that describes the image in length. Empirically they (Ian Hickson in particular) found that the attribute was almost never used in a useful or correct way, rendering it effectively useless. If the discussion had clearly ended at this point, I would have deducted points for those people use advocated longdesc based on bad judgement, but it would not have effected my trust because anyone can mispredict. But the comments just seemed to reinforce the belief that because it should work, that it would work.

The second post was Ian Hickson’s description of using a popular screen reader (JAWS) — you’ll have to dig into the article some, as it’s embedded in other wandering thoughts. In summary, JAWS is a horrible experience, and as an example it didn’t even understand paragraph breaks (where the reader would be expected to pause). What’s the point of semantic markup for accessibility when the most basic markup that is both presentation and semantic (<p>) is ignored? Ian’s brief summary is that if you want to make your page readable in JAWS you’d do better by paying attention to punctuation (which does get read) than to markup. And if you want to help improve accessibility, blind people need a screen reader that isn’t crap.

Months later we started talking a bit about the accessibility of openplans.org. Everyone wants to do the right thing, no? With my trust eroded, I argued strongly that we should only implement accessibility empirically, not based on "best practices". Well, barring some patterns that seem very logical to me, like putting navigation textually at the bottom of the page, and other stuff that any self-respecting web developer does these days. But except for that, if we want to really consider accessibility we should get a tool and use it. But I don’t really know what that tool should be; JAWS is around $1000, all for what sounds like a piece of crap product. We could buy that, even though of course most web developers couldn’t possibly justify the purchase. But is that really the right choice? I don’t know. If we could detect something in the User-Agent string we could see what our users actually use. But I don’t think there’s information there. And I don’t know what people are using. Optimizing for screen magnifiers is much different that optimizing for screen readers.

Another shortcut for accessibility — a shortcut I also distrust — is that to make a site accessible you make sure it works without Javascript. But don’t many screen readers work directly off browsers? Browsers implement Javascript. Do blind users turn Javascript off? I don’t know. If you use no-Javascript as a hint to make the site more accessible, you might just be wasting your effort.

There’s also some weird perspective problems with accessibility. Blind users will always be a small portion of the population. It’s just unreasonable to expect sighted users to write to this small population. Relying on hidden hints in content to provide accessibility just can’t work. Hidden content will be broken, only visible content can be trusted. Admitting this does not mean giving up. As a sighted reader I do not expect the written and spoken word to be equivalent. I don’t think blind listeners lose anything by hearing something that is more a dialect specific to the computer translation of written text to spoken text. (Maybe treating text-to-speech as a translation effort would be more successful anyway?)

A freely available screen reader probably would help a lot as well. I write my markup to render in browsers, not to render to a spec. Anything else is just bad practice. I can’t seriously write my markup for readers based on a spec.

A while back I was reading this post about creating a simple CMS in Grok. Similar to the author over there I often get questions from friends and family who want to build a simple website about what I’d recommend. I still haven’t figured it out. Most of the big names (Plone, Drupal, etc) feel too big and complex and not content-oriented enough. On the other hand, simple systems like Google Homepages are more like HTML editors and don’t really help with managing content.

I have a vision for something much, much simpler; but I’m not sure it’s a realistic vision. There’s always feature creep. If you implement something simple, what happens when someone wants something more complicated? Do you have to throw it out? That sucks. Do you have to implement the new feature? That’s not very simple.

Here’s what I wish existed:

• Just edit content. Static files.
• Probably edit content that is slightly dynamic. Probably PHP code, but without getting too fancy with the PHP. E.g., put <?php include(’header.php’) ?> at the top of each file along with a footer and some variable assignments (like $title).
• Pages would have simple parent/child relationships.
• Sidebar navigation would be mostly manually managed. It would be a single PHP file, like any other. The parent/child relationships would help inform and automate aspects of that generation, but only on the UI side. There would be no attempt to incorporate fancy navigation algorithms.
• The fancy things would be done with Javascript. For instance, if you want to expand navigation depending on the context of the site. This would be easy enough to implement in Javascript, serving everyone the same content and rendering it just slightly differently to emphasize context.
• Of course it would include the important stuff like versioning and staging. With a file-oriented system you could use an existing version control system, or just simple archiving.

At my last job we wrote a content management system, and one of the last features I added I found really clever; a kind of coding judo. We wanted to add structured page types — things like a news feature with dates summary paragraphs. Instead of adding a fancy structured page system (akin to Archetypes in Plone) I just had the system look for an edit form alongside where the page would go (which itself is just a static file). The edit form has the necessary fields, and when you save the page each field turns into a variable assignment. Then there’s a template that uses these assignments. In PHP you’d have something like this:

<? include('header.php'); ?>
  <h1><? echo $page_title; ?></h1>
  Posted on: <? echo $page_date ?><br />
  <blockquote><? echo $page_summary ?></blockquote>
  <div class="article"><? echo $page_article ?></div>
<? include('footer.php'); ?>

The edit form:

Title: <input type="text" name="title"><br />
Date: <input type="text" name="date"> (YYYY-mm-dd)<br />
Summary:<br />
<textarea name="summary"></textarea> <br />
Article:<br />
<textarea name="article"></textarea>

You would use Javascript to make this form fancy if you were so inclined; adding client-side validation (clients were trusted to submit valid data) or marking things for a WYSIWYG editor (usually just with class="wysiwyg" and rely on unobtrusive Javascript to enable it).

For new pages you just give the form. For editing a page you use htmlfill to fill in the form with existing values.

The thing that is saved looks like:

<?
# This page is automatically generated; do not edit directly!
$page_title = 'A Title';
$page_date = '2008-01-11';
$page_summary = 'A summarynetc';
$page_article = 'The Article';
include('article.php');
?>

The system would know how to parse variable assignments (to get values) and would parse the include statement to determine what editor would be used.

Extending this system required a knowledge of HTML and HTML forms, and a little knowledge of PHP (or SSIs would also work). It fit our company well, as there were quite a few people with that kind of knowledge who would interact with clients, and wouldn’t need to come to the programmers to extend the system. While I’d offer some advice on how to use the system, it was transparent enough that they could debug problems on their own.

This whole concept is not something I can justify spending time on, but it’s a CMS that I wish existed. It’s a CMS I could recommend to my friends.

Let’s imagine we live in a late binding world, a programming world of messages, what should that look like? What does it look like in the large?

The Smalltalk notion of this is many, many independent referencable objects. Objects all the way down. So you might get something like buddies do: [ buddy | buddy sendMessage: myMessage].

At runtime this means something like:

• Send the buddies object (some reference we obtained elsewhere) the do: message, with the argument [ buddy | buddy   sendMessage: myMessage]. Except remember that that […] is a block (aka closure), and it’s also an object. Which only makes sense, because buddies doesn’t know what myMessage means. So all buddies knows is that it gets an argument.
• The buddies object receives this do: message, with an argument we’ll call block (that […] stuff). By convention do: is sent to sequence-like things, and calls its argument multiple times. So buddies figures out some items — maybe it’s a concrete sequence (like an Array) or maybe it looks it up in a database, or does a call over the net, or whatever.
• Once buddies gets an item, it calls block value: item. That is, it sends a value: message back to the block object, with the argument item.
• The block has just received that value: message. The block wouldn’t have to be that […] syntactic construct, it just usually is. So it receives that value, and locally binds the buddy variable to the argument and evaluates the inside of that block.
• Our expression then sends the sendMessage: message to that buddy object, with another object as the argument. That argument might be a string, but it also might be markup, or a richer object that itself does all sorts of magical things. Maybe the buddy object sends the object over the net to your remote buddy, then the remote buddy calls message displayOn: myScreen, and the message object reponds to displayOn: by sending screen showText:   someString withColor: #blue. This can go on for a while.

As you can see, there can be a lot of tracing through code in this style. This has been called Ravioli code. Many people lean heavily on concepts like classes and methods to make sense of this kind of code, though if you look through my description you’ll notice they don’t show up; it is apparently somewhat frustrating for Alan Kay that he first used the term "object-oriented" when he would have preferred this message-oriented concept. Classes and methods are just implementation details. Messages and object references are the true fundamental concept.

Doing this in the large means that a lot of these messages go over the network. It means we can get a handle to an object that lives on another computer. It might result in a system that is very chatty (many systems with transparent RPC do end up being unintentionally chatty). But it doesn’t have to. You can make this asynchronous to facilitate these kinds of efficiencies.

That said, there is another distinct technique for programming in the large: document oriented programming. This is what REST is focused on: transmitting state in as complete a form as feasible, with explicit references whenever referring to another resource.

Resources and objects are in many ways similar, but in other ways completely opposite. An object has behavior but no visible state. It responds to messages, passing more objects along. Resources themselves have no behavior. They are just bare state. REST and the web kind of adds behavior, but strict REST avoids even that. PUT is not behavior. PUT is almost like a declaration of fact. When you PUT to a resource you are claiming that the new state of the resource represents the truth of that resource. Any behavior that occurs because of that is hidden. Other resources may change as a result of the PUT, but this is not exposed in any way. To find out what’s changed you simply have to refresh all the state you’ve acquired — re-GET the resources. (POST, except when used simply to create a resource, is more like a verb than a statement of truth; the HTTP model is not pure in this respect.)

For programming on a web scale resources have become the dominant technique. REST was really an acknowledgement of this idea, not the invention of it. It’s also an advocacy effort that our future work should be built on this past success. It’s been focused very clearly on advocating resources over the object/message-based systems of WS-*, but the WS-* design is hardly the first or only object based system of its kind; anything called "RPC" comes from that line of thinking. But now that WS-* isn’t king of the hill anymore maybe we can more be more thoughtful about how REST relates to other programming techniques.

A way to describe the difference in terms of programming language design is that with resources everything we do is call by value. Resources are values, and we pass them around in their complete form. You can pass around a reference value (a URI), but this is like calling a function with a pointer argument. Object/RPC techniques are call by reference (though at some level there are true "values", things like integers and strings, as things would simply be too chatty otherwise).

One interesting place where objects and resources can overlap is in functional programming. Thinking in terms of call by value or call by reference, in a functional programming language these are identical. Because values are not mutable a value and a copy of the value are equivalent. You can also pass an object between systems by reference, in whole, or some lazy compromise where the object is only incrementally communicated between the processes.

Resources are in a sense immutable regardless of your programming language. When you retrieve a resource you cannot modify it in place, as the result is no longer representative of what the resource really is. You can put a new resource in a named location (PUT), and you can copy resources (maybe making an internal copy). Incidentally this is why I think literal document models are best. It would probably be useful to have a native representation of what a "resource" really is in programming languages, though usually this is not done. A resource should really be a document (content type, some bytes, perhaps a cache of parsed representations) plus the name (URI) and the time. Resources aren’t eternal, but given a time and URI there is (in a REST world) a single canonical representation.

Further investigations of functional programming languages would probably be useful here. I believe strict functional languages have interesting metaphors for handling this kind of information; the eternal immutable sense of what a thing is at a point in time, and a way to reconcile the incomplete knowledge of what a resource is right now. I don’t think this is a clear pattern in Erlang, but is in Haskell.

One useful aspect of resources is the ability to spy on the process, to get a snapshot of it, to understand it in a kind of literal way. This is view source in a browser, or just poking around with curl. You can feel a certain confidence that you can get a real sense of what the truth is using these tools. In an object-based system you can only poke around with a process of 20 questions, hoping that there isn’t some hidden truth that you are unaware of. For instance, talking to a proxy that is almost exactly like the original object, but with some small replacement. Of course internally a server can be peculiarly coupled as well, with relationships between the resources that aren’t explicit or obvious. But at least the faces it projects are fully formed.

I still haven’t fully figured out what distinguishes a document model from an object model, but I think the distinction between the two contains some interesting stuff worth exploring.