I attended PloneConf 2008 recently, to talk about Deliverance. But I’ll talk here more about my observations as a relative outsider to that community. (This post is really written for the Plone community — if you aren’t familiar with Plone this post probably won’t make much sense or be very useful.)

One of the ongoing concerns in the Plone community has been the difficulty of attracting and maintaining developer interest in the community, and generally making Plone easier to work with as a developer. It’s been a very successful community for consulting and integration work, but it has not been as rewarding for developers. There’s the idea of a "Plone Tax", which means different things to different people but just generally speaks to the sense that Plone takes a little off the top — time to restart, time to run the tests, time to dig into code that just goes too deep. There are some distinct problems with Plone, but probably the biggest problem is the quantity of small challenges and the general size and complexity of the system.

At the moment there is no clear path forward to resolve this. A previous effort to fix things is a project called "Five", which referred to Zope 2+3 — backporting libraries and techniques from Zope 3 into Zope 2. Plone is intimately tied to Zope 2, and Five let them use new code without having to do abandon the old environment. But the result wasn’t terribly satisfying: Five didn’t remove any complexity, it only added to it. Even if the new components were superior that doesn’t make them simple. Plone is aching for more simplicity, not more power.

It’s unclear how Plone can actually reform itself as a codebase. Zope 2 is a behemoth, and its metaphors are deeply intertwined with existing code. Acquisition in particular is ubiquitous, essential to lots of the machinery, and deeply confusing. I saw a general interest in two directions: one was to encourage non-content-management tasks to be implemented in a complementary (but separate) technology, another direction is continuing to refactor the existing codebase while somehow trying to maintain backward compatibility.

It’s unclear how to refactor the existing codebase in such a way that it is any simpler, but I suppose these two directions are not exclusive. I want to focus on the idea of a complementary environment. There’s two products in particular that have been attracting interest: Grok and Repoze.

These two products usually go under the umbrella of "exciting new ways to improve the developer experience in Plone" which is a kind of generic positive sentiment. But to my mind they represent two significantly different paths forward, and the differences deserve some more critical thought.

Grok is a layer on top of Zope 3 that attempts to make development in that environment more pleasing. It eliminates most ZCML (ZCML is Zope 3’s XML-based language for declaring the relation of various components in a system). Grok uses conventions and introspection to make Zope 3 look more like a traditional web framework, with simple views and models and templates, and less of the wiring you have to set up in typical Zope 3 architectures. At the same time, you can add all the Zope 3 declarations to break out of the automatic conventions. Grok is a more pleasant layer on top of Zope 3, but it’s entirely focused on Zope 3, and it is led by Martijn Faassen and Philipp von Weitershausen who are both very involved with the Zope 3 community.

Repoze is a more recent project, led by Chris McDonough, Tres Seaver, and Paul Everitt (well, Paul might call himself more of a cheerleader). They all work together at Agendaless Consulting. They’ve been highly involved in Zope 2 for a long time, and are all former employees of Zope Corp and major contributors to Zope 2. I don’t think they’ve ever quite made the jump to Zope 3, and their consulting and experience kept them involved with Zope 2. A while back they got some WSGI religion and started splitting out some pieces of Zope into WSGI middleware and other independent libraries. This included things like pulling out the transaction handler from Zope, reimplementing the Zope 2 publisher so it was more WSGIish, and a variety of other libraries. These libraries are essentially extractions from Zope, or in the case of the Zope publisher and repoze.plone, a way to wrap what would be considered a "legacy" application in the same interface as other newer pieces. Having extracted nearly everything they wanted, they’ve started work on a framework intended to be familiar to the Zope community, repoze.bfg. This framework uses some Zope 3 concepts, but it’s more built from scratch than it is built from Zope 3, and it is attached to Zope 2 only insofar as it uses the pieces they’ve extracted and the ideas they’ve become comfortable with. They’ve described it as the framework they want to use when someone asks them to build something in Zope 2.

Grok and Repoze have a significantly different development methodology. One is a layer on Zope 3, the other is an extraction of ideas from Zope 2 (and a few from Zope 3). In part I think the distinction hasn’t been presented very clearly because the Repoze and Grok communities overlap a great deal, and everyone is quite congenial with each other, and they are reluctant to enter debates about the designs. While I also consider them all colleagues, I also feel pretty strongly about the design differences and I feel a discussion contrasting them is important.

Martijn recently wrote a post on why Plone should consider Grok and a follow-up post. These posts speak to a variety of advantages Grok has over plain Zope 3, which Grok can offer to Plone to manage its existing use of Zope 3 technologies.

I think Plone shouldn’t be so focused on managing the complexity of its stack, but focus on reducing that complexity. And it should reduce that complexity by focusing on content management and moving all the other pieces people have built on Plone out of Plone. It can’t just leave people hanging, which is why developing a clear story for how those other pieces should be developed is essential. Plone the community doesn’t have to map one-to-one to Plone the software. Plone the software should become smaller and more focused. Plone the community doesn’t have to become more focused — the community does what it needs to do, what customers ask for, what is necessary to make a site compelling. To be clear on what I’m proposing here: Plone should have a community-recommended way to build non-Plone applications. This doesn’t mean you have to use those techniques, but it should be much more concrete than just "there’s a bunch of cool things out there, and maybe you should look around and use one of those." By having a community-recommended pattern of development you can maintain and build on the Plone community, which is at least as important an asset as the Plone software.

With this in mind I believe an extraction of Zope 2 and Plone ideas is the right path forward. Extraction is the process of isolating code and ideas, and localizing the effect of that code. These are some of the most important ways to actually increase the simplicity of a codebase. In the Zope 2 (and even 3) codebase the thing that brings the most complexity is the non-locality of code, that parts of the system can effect each other in complex and unexpected ways. Zope 3 formalizes these patterns of non-locality: the Component Architecture is largely about introducing non-localized relationships between code.

Arguably this non-locality of effect is exactly what people want, it’s what enables pervasive customizations. When a client asks you to change some little piece deep in the system, you don’t really want to modify the system’s code — you want to add a little code to the outside of the system that effects the change you desire. The Component Architecture is a formalized way of making these kinds of changes, where Acquisition was a lower-level mechanism to do the same sort of thing. I remain a Component Architecture skeptic, mostly because I think the mechanism is overused. Still I think clearly Plone needs flexibility that a purely bespoke application would not require. But there’s no way out of it: that flexibility has a high cost. In this there is no clear solution. In addition to Acquisition, the complexity of Zope 2 security, the many layers of skinning… is the Component Architecture what Plone needs? From where I stand it seems like a step further in the wrong direction… and yes, it is a better placed step than any of the ones before, but if it’s still a step in the wrong direction does that matter?

I think Plone (at least the community) needs to be conservative in its enabling of these customizations. Sure, the customizability is "powerful," but I don’t hear people clamoring for power. They want simple, predictable, fast, maintainable. That’s not necessarily the same as "easy" — I think sometimes it’s worth making things a little longer and less automatic to make a system more explicit and make code more localized. I think Repoze is a step in this direction, a big (maybe even an intimidatingly giant) step towards simplicity. That’s the step I think Plone should make.

So, I offer this as my suggestion to Plone. I think Plone-the-community has the opportunity to be more than Plone-the-software; I think it must do this to remain viable in the long term. But to get there the community make some choices — you can’t add simplicity.

I’ve finished renaming pyinstall to its new name: pip. The name pip is a acronym and declaration: pip installs packages.

I’ve also added a small feature intended for Google App Engine users, allowing you to zip and unzip packages in an environment. For instance:

$ pip zip --list
In ./lib/python2.5/site-packages:
  No zipped packages.
  Unzipped packages:
    paste  (98 files)
    pygments  (64 files)
    tempita  (7 files)
    weberror  (31 files)
    webob  (22 files)
    webtest  (9 files)
    nose  (43 files)
    setuptools-0.6c9-py2.5.egg  (43 files)
    simplejson  (28 files)
$ pip zip webob
Zip webob (in ./lib/python2.5/site-packages/webob)

Right now this doesn’t work well with egg directories (i.e., packages installed with easy_install), though that shouldn’t be too hard to resolve. pip install itself does not install packages into egg directories (it does install eggs, which is to say it installs all the egg metadata and works fine with pkg_resources).

I don’t really use buildout myself, but I would like to throw it out there that I think someone should create a pip recipe as an alternative to zc.recipe.egg. There’s not really a stable programmatic API in pip at this point, but with no consumers of the API it feels like premature design to settle on something now — integrate with pip and we can figure out what that stable API should be. If you integrate buildout, probably another useful feature would be an option have to pip freeze write the packages out to a setting in your buildout.cfg.

Roy T. Fielding, author of the REST thesis wrote an article recently: REST APIs must be hypertext-driven. I liked this article, it fit with an intuition I’ve had. Then he wrote an article explaining that he wouldn’t really explain the other articles because, I guess, he wanted a conversation with the specialists, and it seems like a kind of invitation to reinterpret his writing. So since others are doing it I figured I’d do it too.

I’d summarize his argument thus:

• Focus on media types, i.e., resource formats, i.e., document formats. The protocol will flow from these if they are well specified.
• URL structures are not a media type. They are some kind of server layout. You can’t hold them, you can’t pass them around, there is no notion of CRUD. Media types have all sorts of advantages that URL structures do not.

An example of a protocol based on a URL structure would be something like:

• Do GET /articles/ to get a JSON list of all the article ids, with a response like [1, 2, 3]
• Do a GET /articles/{id} to get the representation of a specific article.

JSON is a reasonable structure for a media type. It is not itself a fully explained type, because it’s just a container for data, just like XML. In this example you have a document, [1, 2, 3] which isn’t self-describing and just isn’t very useful. A more appropriate protocol would be:

• You start with a container, in our example /articles/. Do GET /articles/ to get a JSON document listing the URLs of all the articles. These URLs are relative to the container URL. You’ll get a response like [’./1′, ‘./2′, ‘./3′] (actually [’1′, ‘2′, ‘3′] would be fine too).
• Do GET {article-url} to get the article representation.

It’s a small difference. Heck, the communication could look identical in practice, but by putting URLs in the JSON document instead of this abstract "id" notion you’ve created a more flexible and self-describing system. You could probably give a name to that list of URLs, and then just talk about that name.

An example in Atompub is rel="edit". An Atom entry can look like:

<entry>...
  <link rel="edit" href="/post/15" />
</entry>

Instead of the client just somehow knowing where to go to edit an entry, it’s made explicit. Thus you can move the entry around, while still pointing back to the canonical location to edit that entry.

There’s nothing really that complicated about this, the rule is really quite simple: link to other things, don’t just expect the client to know or guess where those other things are.

For a more concrete example of where this linking works well, OpenID uses <link rel="openid.server" href="…"> and <link rel="openid.delegate" href="…">, which allows you to add a little information to any HTML homepage so that the login can happen at a third location. If OpenID used something like looking at {homepage}/openid for a OpenID server then you couldn’t select whatever OpenID service you liked, or change services, or apply OpenID to hosted locations where you couldn’t install an OpenID server.

I’ll add my own little opinion in here: this is why the URL structure of applications doesn’t affect their RESTfulness, nor is URL structure all that important of a concern generally. Pretty URL structures are a nice thing to do, like indenting your code in a pleasant way, but it has nothing to do with your API, and if you can’t use a crappy URL structure with that same API then probably something is wrong with that API.

So, decorators are neat (maybe check out a new tutorial on them). Descriptors are neat but may seem hard (though they hardly take a long time to describe). Sometimes these two things intersect, and this post describes how.

Here’s an example decorator where this comes up. First, we want something that looks like a WSGI application:

def application(environ, start_response):
    start_response('200 OK', [('content-type', 'text/html')])
    return ['hi!']

But we want to use WebOb, like:

from webob import Request, Response
def application(environ, start_response):
    req = Request(environ)
    resp = Response('hi!')
    return resp(environ, start_response)

(We don’t use req in this example, but of course you probably would in a real WSGI application)

Now req = Request(environ) is boilerplate, and it’d be nicer to just do return resp instead of return resp(environ, start_response). So let’s make a decorator to do that:

class wsgiapp(object):
    def __init__(self, func):
        self.func = func
    def __call__(self, environ, start_response):
        resp = self.func(Request(environ))
        return resp(environ, start_response)

@wsgiapp
def application(req):
    return Response('hi!')

If you don’t understand what happened there, go read up on decorators.
Now, what if you want to decorate a method? For instance:

class Application(object):
    def __init__(self, text):
        self.text = text
    @wsgiapp
    def __call__(self, req):
        return Response(self.text)
application = Application('hi!')

This won’t quite work, because @wsgiapp will call Application.__call__(req) — with no self argument. This is generally a problem with any decorator that changes the signature, because the signature for methods has this extra self argument. Descriptors can handle this. First we’ll have the same wsgiapp definition as we had before, but we’ll add the magic descriptor method __get__:

class wsgiapp(object):
    def __init__(self, func):
        self.func = func
    def __call__(self, environ, start_response):
        resp = self.func(Request(environ))
        return resp(environ, start_response)
    def __get__(self, obj, type=None):
        if obj is None:
            return self
        new_func = self.func.__get__(obj, type)
        return self.__class__(new_func)

So, to explain:

When you get an attribute from an instance, like Application().__call__, Python will check if the object that was fetched has a __get__ method. If it does, it will call that method and use the result of that method.

This part:

if obj is None:
    return self

is what happens when you do Application.__call__ — in other words, when get a class attribute. In that case obj (self) will be None, and it will just return the descriptor (it could do something else, like in this example, but usually it doesn’t).

Functions already have a __get__ method. You can try it yourself:

>>> def example(*args):
...     print 'got', args
>>> example_bound = example.__get__(1)
>>> example_bound('test')
got (1, 'test')

So in the example with wsgiapp we are just changing the decorator to wrap the new bound function instead of the old unbound function. This allows wsgiapp to be compatible with both plain functions and methods. In fact, it would probably be preferable to always call func.__get__(obj, type) (even if obj is None), as then we could also wrap class methods or other kinds of descriptors.

I’ll be attending PloneConf this year again, giving a talk about Deliverance. I’ve been working on Deliverance lately for work, but the hard part about it is that it’s not obviously useful. To help explain it I wrote the philosophy of Deliverance, which I will copy here, to give you an idea of what I’ve been doing:

Why is Deliverance? Why was it made, what purpose does it serve, why should you use it, how can it change the way you do web development?

We had a debate party tonight for the Biden-Palin debate. It’s nice to watch it in a group of like-minded people. Taking the Democrat/Republican debate seriously is a bullshit game and I don’t have any desire to bring this farce into my normal life.

After the debate was over, I wanted to discuss the debate. After all, it’s weird to watch something for an hour and a half and then just ignore that we spent that time watching it. The problem is that I hate the punditry. No one actually said "did Palin do what she had to do?" (I probably would have screamed) but it’s just really hard not to talk about "what will people think of this debate?" And part of that is because we all know what we think. We saw through Palin deliberately ignoring the questions and reading her already-prepared speech. We all had a basic understanding of what is fact and what is a lie or misrepresentation. It’s nice to share little stories (like stories from the article about how McCain is a jerk). But it’s so damn hard not to fall into a discussion about the horserace, about what other people will think. Why is it so hard to talk about what we think? Not what we analyze, but what we actually believe? Instead of predicting something that will come to pass regardless of our predictions, shouldn’t we be developing our own beliefs? That seems far more relevant to our lives.

There’s probably a lot of reasons for that. It’s intimidating to be entirely genuine, to speak without irony. And all the news is about the horserace, so we are all well informed, it makes it easy to talk.

I think a large part of the problem is that the spectrum of opinions is so narrow (even if also bifurcated) that it’s hard to have an interesting discussion of political issues. Lacking anything of real substance to discuss, we discuss the discussion, we make predictions instead of forming real opinions. While I’m willing to blame many things on the Republicans, this is the product of both parties, of the narrow ignorance of "conventional wisdom." For instance, the debate about the economic bailout has been rich with rhetoric but starved of any real ideas. I didn’t even realize how limited the debate was until I listened to this interview where Steve Fraser kind of says, well, we can do whatever we want. That is to say, we can actually make collective decisions about the direction of our economy, instead of the impotent position that is assumed in all current debates, where we can only poke lightly at the economy (and it’s implied anything more would destroy it).

We can’t really talk about what kind of healthcare system we’d like, because the system nearly everyone wants is not an acceptable part of conventional wisdom. Socialized healthcare is the only reasonable option, but of course there’s lots of ways it could work, there’s lots of room for genuine and important discussion. But instead we have a staggeringly horrible proposal, and a merely not quite as bad as the current situation proposal. Given this set of options you can’t have real discussion.

In the end our own happiness is mostly in our own hands. The choices we make for ourselves are more significant than the choices made by the government (the choices we make collectively). But our collective choices do matter. We certainly haven’t figured out happiness. And maybe government does best when it has the least effect on our lives, but while that’s one end of the bifurcated conventional wisdom, as an idea it remains largely uninspected. When I consider many of the pleasant conveniences in my life, government is part of a lot of them. It doesn’t do much to make me more spiritually fulfilled, but the idea that government is a hopeless place to look for our collective happiness is a truism that lacks real consideration.

Political discussion is stuck in a terrible intellectual rut. Blame falls equally on both parties. They hold on greedily to their monopoly of political thought. It’s like religious doctrine, something to which politicians must submit before being allowed to progress, a sign of submission to a larger system of power. I have this hope that Obama is going through the rites with discipline but without true belief, that he is being subversive, diving straight to the belly of the beast. But this is only speculation, perhaps a naive dream, a desire to project my hopes onto a figure of vague and general hope.

I don’t really want to spend too much time discussing all the things that are wrong. This is the depressing comfort zone of the left. I want to talk about how things could be right, about how we can make a world that isn’t just less unjust but a world that is more beautiful, more wonderful, more full of life and freedom and passion. I want to exult in the potential of the future.

Update: pyinstall has been renamed to pip (per the suggestions in the comments to this post)

I added pybundles to pyinstall very shortly before I announced pyinstall. I hadn’t actually tried it out that much. Since then I’ve made three more minor releases of pyinstall, and I think the bundle support is working pretty decently.

A .pybundle file is just a bunch of source code, all the source code you need to install some package(s). For instance, for Deliverance I’ve created a bundle file so you can do:

$ easy_install pyinstall virtualenv
$ pyinstall.py -E DeliveranceTest/ \
>   http://deliverance.openplans.org/dist/Deliverance-snapshot-latest.pybundle

This creates a virtualenv environment in DeliveranceTest/, unpacks all the source from the bundle, and installs all of it. It’s not magical — it still has to compile the source and move the files around — but it does mean just a single download, and the versions of everything that is installed aren’t going to change unless that bundle file is regenerated.

I’ve been thinking about some other features for pybundles, like post installation scripts. All of this has raised a problem though: pyinstall needs to be a two-level command, with commands like:

pyinstall.py install X
pyinstall.py bundle Y
pyinstall.py freeze req.txt
and of course the not-yet-implemented:
pyinstall.py remove Z

But pyinstall.py install does not read well. It’s not too late to rename the package (yet again), or just rename the script. Ideas?

Note: pyinstall has been renamed to pip

Ever been frustrated with easy_install? Interested in an alternative? Then you should go over to my work blog and read pyinstall: A New Hope.

I was talking with a coworker some time ago about his project, and he needed to update a piece of the page in-place when you go back to the page, and setting the page as uncacheable didn’t really work. Which probably makes sense; I think at one time browsers did respect those cache controls, but as a result going back in history through a page could cause some intermediate page to be refreshed and needlessly slow down your progress.

Anyway, Rails uses partials to facilitate this kind of stuff in a general way. Bigger chunks of your page are defined in their own template, and instead of rendering the full page you can ask just for a chunk of the page. Then you do something like document.getElementById(’some_block’).innerHTML = req.responseText. Mike Bayer just described how to do this in Mako too, using template functions.

When asked, another technique also occurred to me, using just HTML. Just add a general way of fetching an element by ID. At any time you say "refresh the element with id X", and it asks the server for the current version of that element (using a query string variable document_id=X) and replaces the content of that element in the browser.

The client side looks like this (it would be much simpler if you used a Javascript library):

function refreshId(id) {
    var el = document.getElementById(id);
    if (! el) {
        throw("No element by id '" + id + "'");
    }
    function handler(data) {
        if (this.readyState == 4) {
            if (this.status == 200) {
                el.innerHTML = this.responseText;
            } else {
                throw("Bad response getting " + idURL + ": "
                      + this.status);
            }
        }
    }
    var req = new XMLHttpRequest();
    req.onreadystatechange = handler;
    var idURL = location.href + '';
    if (idURL.indexOf('?') == -1) {
        idURL += '?';
    } else {
        idURL += '&';
    }
    idURL += 'document_id='+escape(id);
    req.open("GET", idURL);
    req.send();
}

Then you need the server-side component. Here’s something written for Pylons (using lxml.html, and Pylons 0.9.7 which is configured to use WebOb):

from pylons import request, response
from lxml import html

def get_id(response, id):
    if (response.content_type == 'text/html'
        and response.status_int == 200):
        doc = html.fromstring(response.body)
        try:
            el = doc.get_element_by_id(id)
        except KeyError:
            pass
        else:
            response.body = html.tostring(el)
    return response

class BaseController(WSGIController):
    def __after__(self):
        id = req.GET.get('document_id')
        if id:
            get_id(response, id)

Though I’m not sure this is appropriate for middleware, you could do it as middleware too:

from webob import Request
class DocumentIdMiddleware(object):
    def __init__(self, app):
        self.app = app
    def __call__(self, environ, start_response):
        req = Request(environ)
        id = req.GET.get('document_id')
        if not id:
            return self.app(environ, start_response)
        resp = req.get_response(self.app)
        resp = get_id(resp, id)
        return resp(environ, start_response)

Saturday morning my sister, Monica Bicking, and her boyfriend, Eryn Trimmer, were arrested in Minneapolis. Monica was released on Sunday, but Eryn and others are still in custody, and the police will try to keep them detained as long as possible. update: the two of them and six others from the Welcoming Committee are charged with felonies, including "furtherance of terrorism". A website has been set up in support of them, and to keep people informed about ongoing events in the case: rnc8.org

They were arrested for "conspiracy to incite a riot". This is the same charge used against the Chicago 8 (or 7) at the 1968 Democratic Convention. Perhaps the police have a sense of tradition?

But more directly she and Eryn were arrested in an attempt to preemptively suppress the protests at the Republican National Convention. They were both very active with the RNC Welcoming Committee, which is a group coordinating and supporting some of the people coming to the Twin Cities for the convention.

Obviously I’m very concerned by the arrests and charges. But there’s been a huge outpouring of support from the community — both from activist in the Twin Cities, and from their neighbors. In Chicago I’m a little unsure about what to do.

Reading articles about the incidents (Glenn Greenwald’s post on Salon is a good one) I find myself mostly avoiding the comment sections. The comments fall into two categories: mean comments against the protesters, and reactionary comments with no real substance ("this is proof this country is a police state!") Activists generally understand what’s going on, and people of a right-wing/authoritarian bend are hardly going to be convinced of anything, but there’s a lot of progressive people out there who’ve never really been involved in any activism like this. There’s very little explaining the protests, the role of activists like my sister, and the philosophies they hold. Certainly the news makes no attempt, and unfortunately the activists themselves often speak from an unexplained perspective.

So I’d like to use this as an opportunity to explain my understanding of the role of protest, what’s going on at the RNC specifically, and what an "anarchist" really is. At the moment I can’t do a lot to help Eryn and Monica directly, but at least I can talk about her personally instead of another story about a named but otherwise anonymous "protester".