Ian Bicking: a blog » Search Results » pylons

WebTest HTTP testing

Ian Bicking — Fri, 02 Apr 2010 22:27:37 +0000

I’ve yet to see another testing system for local web testing that I like as much as WebTest… which is perhaps personal bias for something I wrote, but then I don’t have that same bias towards everything I’ve written. Many frameworks build in their own testing systems but I don’t like the abstractions — they touch lots of internal things, or skip important steps of the request, or mock out things that don’t need to be mocked out. WSGI can make this testing easy.

There’s also a hidden feature here: because WSGI is basically just describing HTTP, it can be a means of representing not just incoming HTTP requests, but also outgoing HTTP requests. If you are running local tests against your application using WebTest, with just a little tweaking you can turn those tests into HTTP tests (i.e., actually connect to a socket). But doing this is admittedly not obvious; hence this post!

Here’s what a basic WebTest test looks like:

from webtest import TestApp
import json

wsgi_app = acquire_wsgi_application_somehow()
app = TestApp(wsgi_app)

def test_login():
resp = app.post('/login', dict(username='guest', password='guest'))
resp.mustcontain('login successful')
resp = resp.click('home')
resp.mustcontain('guest')
# Or with a little framework integration:
assert resp.templatevars.get('username') == 'guest'

# Or an API test:
def test_user_query():
resp = app.get('/users.json')
assert 'guest' in resp.json['userList']
user_info = dict(username='guest2', password='guest2', name='Guest')
resp = app.post('/users.json', content_type='application/json',
body=json.dumps(user_info)
assert resp.json == user_info

The app object is a wrapper around the WSGI application, and each of those methods runs a request and gets the response. The response object is a WebOb response with several additional helpers for testing (things like .click() which finds a link in HTML and follows it, or .json which loads the body as JSON).

You don’t have to be using a WSGI-centric framework like Pylons to use WebTest, it works fine with anything with a WSGI frontend, which is just about everything. But the point of my post: you don’t have to use it with a WSGI application at all. Using WSGIProxy:

import os
import urlparse

if os.environ.get('TEST_REMOTE'):
from wsgiproxy.exactproxy import proxy_exact_request
wsgi_app = proxy_exact_request
parsed = urlparse.urlsplit(os.environ['TEST_REMOTE'])
app = TestApp(proxy_exact_request, extra_environ={
'wsgi.scheme': parsed.scheme,
'HTTP_HOST': parsed.netloc,
'SERVER_NAME': parsed.netloc})
else:
wsgi_app = acquire_wsgi_application_somehow()
app = TestApp(wsgi_app)

It’s a little crude to control this with an environmental variable ($TEST_REMOTE), but it’s an easy way to pass an option in when there’s no better way (and many test runners don’t make options easy). The extra_environ option puts in the host and scheme information into each request (the default host WebTest puts in is http://localhost). WSGIProxy lets you send a request to any host, kind of bypassing DNS, so SERVER_NAME is actually the server the request goes to, while HTTP_HOST is the value of the Host header.

Going over HTTP there are a couple features that won’t work. For instance, you can pass information about your application back to the test code by putting values in environ['paste.testing_variables'] (which is how you’d make resp.templatevars work in the first example). It’s also possible to use extra_environ to pass information into your application, for example to get your application to mock out user authentication; this is fairly safe because in production no request can put those same special keys into the environment (using custom HTTP headers means you must carefully filter requests in production). But custom environ values won’t work over HTTP.

The thing that got me thinking about this is the work I’m doing on Silver Lining, where I am taking apps and rearranging the code and modifying the database configuration ad setup to fit this deployment system. It would be really nice having done that to be able to run some functional tests, and I really want to run them over HTTP. If an application has tests using something like Selenium or Windmill that would also work great, but those tools can be a bit more challenging to work with and applications still need smaller tests anyway, so being able to reuse tests like these would be most useful.

What Does A WebOb App Look Like?

Ian Bicking — Fri, 12 Mar 2010 23:50:30 +0000

Lately I’ve been writing code using WebOb and just a few other small libraries. It’s not entirely obvious what this looks like, so I thought I’d give a simple example.

I make each application a class. Instances of the class are "configured applications". So it looks a little like this (for an application that takes one configuration parameter, file_path):

class Application(object):
def __init__(self, file_path):
self.file_path = file_path

Then the app needs to be a WSGI app, because that’s how I roll. I use webob.dec:

from webob.dec import wsgify
from webob import exc
from webob import Response

class Application(object):
def __init__(self, file_path):
self.file_path = file_path
@wsgify
def __call__(self, req):
return Response('Hi!')

Somewhere separate from the application you actually instantiate Application. You can use Paste Deploy for that, configure it yourself, or just do something ad hoc (a lot of mod_wsgi .wsgi files are like this, basically).

I use webob.exc for things like exc.HTTPNotFound(). You can raise that as an exception, but I mostly just return the object (to the same effect).

Now you have Hello World. I then sometimes do something terrible, I start handling URLs like this:

@wsgify
def __call__(self, req):
if req.path_info == '/':
return self.index(req)
elif req.path_info.startswith('/view/'):
return self.view(req)
return exc.HTTPNotFound()

This is lazy and a very bad idea. So you want a dispatcher. There are several (e.g., selector). I’ll use Routes here… the latest release makes it a bit easier (though it could still be streamlined a bit). Here’s a pattern I think makes sense:

from routes import Mapper

class Application(object):
map = Mapper()
map.connect('index', '/', method='index')
map.connect('view', '/view/{item}', method='view')

def __init__(self, file_path):
self.file_path = file_path

@wsgify
def __call__(self, req):
results = self.map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(self.map, req.environ)
req.urlvars = ((), match)
kwargs = match.copy()
method = kwargs.pop('method')
req.link = link
return getattr(self, method)(req, **kwargs)

def index(self, req):
...
def view(self, req, item):
...

Another way you might do it is to skip the class, which means skipping a clear place for configuration. I don’t like that, but if you don’t care about that, then it looks like this:

def index(self, req):
...
def view(self, req, item):
...

map = Mapper()
map.connect('index', '/', view=index)
map.connect('view', '/view/{item}', view=view)

@wsgify
def application(req):
results = map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(map, req.environ)
req.urlvars = ((), match)
kwargs = match.copy()
view = kwargs.pop('view')
req.link = link
return view(req, **kwargs)

Then application is pretty much boilerplate. You could put configuration in the request if you wanted, or use some other technique (like Contextual).

I talked some with Ben Bangert about what he’s trying with these patterns, and he’s doing something reminiscent of Pylons controllers (but without the rest of Pylons) and it looks more like this (with my own adaptations):

class BaseController(object):
special_vars = ['controller', 'action']

def __init__(self, request, link, **config):
self.request = request
self.link = link
for name, value in config.items():
setattr(self, name, value)

def __call__(self):
action = self.request.urlvars.get('action', 'index')
if hasattr(self, '__before__'):
self.__before__()
kwargs = req.urlsvars.copy()
for attr in self.special_vars
if attr in kwargs:
del kwargs[attr]
return getattr(self, action)(**kwargs)

class Index(BaseController):
def index(self):
...
def view(self, item):
...

class Application(object):
map = Mapper()
map.connect('index', '/', controller=Index)
map.connect('view', '/view/{item}', controller=Index, action='view')

def __init__(self, **config):
self.config = config

@wsgify
def __call__(self, req):
results = self.map.routematch(environ=req.environ)
if not results:
return exc.HTTPNotFound()
match, route = results
link = URLGenerator(self.map, req.environ)
req.urlvars = ((), match)
controller = match['controller'](req, link, **self.config)
return controller()

That’s a lot of code blocks, but they all really say the same thing ;) I think writing apps with almost-no-framework like this is pretty doable, so if you have something small you should give it a go. I think it’s especially appropriate for applications that are an API (not a "web site").

Toward a new self-definition for open source

Ian Bicking — Fri, 11 Sep 2009 00:09:11 +0000

This is roughly the speech I gave as a keynote address at DjangoCon 2009 in Portland.

I’ve been programming Python web software for quite a while now. I considered coming here and talked about WSGI, standards, cross-framework integration, etc., but I decided I wanted to come up here and talk to you as compatriots, as fellow open source programmers.

Over the past year or so I have been thinking a lot about politics. Not electoral politics per se, or the geopolitical situation, but the question of the meaning of what we are doing. Maybe it is some sort of programmer midlife crisis: why am I doing what I’m doing, and why does it matter? And also I have been thinking a lot about open source — what this thing that we’re doing means in the larger context. Are we just engineers? Is there some sort of movement? If so, what is that movement? Especially as open source has become more popular, the sense of a movement seems to dwindle. It felt like a movement 10 years ago, but not as much today. Why should this happen? Why now, in the midst of success does open source seem less politically relevant than it did 10 years ago?

I’m also speaking somewhat to Django specifically, as I think it is one of the communities with a bit more resistance to the idea of the politics of code. The Django philosophy is more: the value of this code is the value of what you do with it. I’m not here to criticize this perspective, but to think about how we can find meaning without relying on the traditional free software ideas. To see if there’s something here that isn’t described yet.

I’d like to start with a quick history of free and open source software. My own start was in highschool where I was taking a class in which we all used Emacs. This was my first introduction to any real sort of programming environment, to Unix, to a text editor that was anything to talk about. At the time Emacs would say at startup "to learn more about the GNU project hit Control-H Control-P" — because of course you need a keyboard shortcut to get to a philosophy statement about an editor. So one day I hit Control-H Control-P. I was expecting to see some sort of About Message, or if you remember the software of the times maybe something about Shareware, or even "if you really like this software, consider giving to the Red Cross." But instead I came upon the GNU Manifesto.

GNU Manifesto

I’d like to read a couple quotes from the GNU Manifesto. There are more modern descriptions of GNU, but this is one of the first documents describing the project and its mission, written by Richard Stallman. Let me quote the section "Why I Must Write GNU":

"I consider that the golden rule requires that if I like a program I must share it with other people who like it. Software sellers want to divide the users and conquer them, making each user agree not to share with others. I refuse to break solidarity with other users in this way. [it continues...]

"So that I can continue to use computers without dishonor, I have decided to put together a sufficient body of free software so that I will be able to get along without any software that is not free."

[later it goes on...]

"The fundamental act of friendship among programmers is the sharing of programs; marketing arrangements now typically used essentially forbid programmers to treat others as friends. The purchaser of software must choose between friendship and obeying the law. Naturally, many decide that friendship is more important. But those who believe in law often do not feel at ease with either choice. They become cynical and think that programming is just a way of making money."

When I read this statement I was immediate head-over-heels in love with this concept. As a teenager, thinking about programming, thinking about the world, having a statement that was so intellectually aggressive was exciting. It didn’t ask: "how wrong is piracy really", or "why are our kids doing this", but it asked "is piracy a moral imperative" — that’s the kind of aggressive question that feels revolutionary and passionate.

Let me go over one of the lines that I think exemplifies this:

"I consider that the golden rule requires that if I like a program I must share it with other people who like it."

It wasn’t saying: what are we not allowed to do, nor did it talk about some kind of societal injustice. It didn’t talk about the meaning of actions or their long-term effects. Instead it asked: what must we do, not as a society, not in service of some end, but what are we called upon to do as an individual, right now, in service of the people we call friends. It didn’t allude to any sociological explanation, natural selection, economics; there is just the golden rule, the most basic tenant of moral thought.

Free Software vs. Open Source

When I first encountered free software, I suppose about 15 years ago, this was during one of the more difficult periods of its evolution. It was past the initial excitement, the initial optimism that the project would take only a couple years to reach parity with closed-source Unix, it was before it was clear how the project would move forward. Linux was young and seemed to be largely off the radar of Richard Stallman and other GNU advocates, and they were struggling to fill in final key pieces, things like a kernel, widgets, and they hadn’t even thought about entirely new things like browsers.

The message that came through from that period is not the message I wish came through. The message I wish came through was that message from the GNU Manifesto, that spirit of a new sense of duty and ability. When people talk about Richard Stallman and the Free Software Foundation and the GNU Project, they’ll often point to the GNU General Public License as the most important contribution — the GPL. I’m sure you all know something about it. Of course the core concept there is the idea of copyleft. Not only will the software be open, but it’s takes the implied perspective that the principles of freedom are rights — that unfortunately the world is not wise enough to see that use, distribution, and modification are rights; but the GPL still asserts that these are your rights. When you become one of us, when you participate in the freedoms this license grants you, when you use the GPL, there is encoded in the license a support for this sense of the natural right of free software. We, GNU, can’t encode that for the world, but for the software that we write these rights are inalienable.

But as I said those were difficult times. There was a great deal of pressure. People were trying to understand what open source meant. People still struggle with questions: how would an economy function, how would a programmer get a job, if this is as successful as people hoped will we all just be out of jobs? Other questions were: who will write the software that no one wants to write? How can I, embedded in a situation where I can’t actually use only free software — remember at this time there was no way to use completely free software — how can I assert a duty to do something that is not possible? How can I be useful unless I interface with all these proprietary things? If I deal with companies which aren’t comfortable with open source software, then what? After all, open source seemed only barely plausible at this time. It was not an easy argument to make.

And all this was before the term "open source" really took hold as a distinct idea. That itself is an interesting story. There was a time during this marketing period when there arose a kind of terminology divide — free software vs. open source software. The terminology divide was that the "free" in free software implied you couldn’t charge anything, that made people think about price, might even imply cheapness. Open source directly refers to programming, uses the feel-good term "open", and doesn’t sound too revolutionary. But besides that there was also a substantial philosophical difference about the value of the software itself.

So there was a difference in how things were going to be pitched, but also a difference in what people thought the general value of this project was. From GNU and Richard Stallman there was the notion that this was right because it was right; it was a moral imperative. The virtue of what we build is in its freedom; if it is also technically superior then that’s great, but it is not how we should judge our success. We were giving people self-determination: programmer self-determination, user self-determination… on the open source side the argument was that this is a good way to create software. Programmers working together can do better work. With many eyes all bugs are shallow. All working together, we’ll work faster, you get the benefit of free contributions from all sorts of people. People were clamouring to get all these proprietary companies with failing software products to open source their software; miracles will occur! What you thought was useless will regain value! You’ll reattain relevance!

The open source and free software philosophical divide: on one side practical concerns, on the other moral. And this is what I want to talk about later: can we find a moral argument for these practical concerns?

The basic free/open disagreement continues in debates over licenses: the GPL vs. the BSD and other permissive licenses. If you read the GPL it talks a great deal about philosophy; if you read the BSD license it’s really just some disclaimers and basic instructions, and the one line: "Redistribution and use in source and binary forms, with or without modification, are permitted." It doesn’t say why you’ve received this software, or any philosophy about rights and freedoms, or even encourage you to use the same license on your own software. An engineer’s license.

So these two licenses in many ways became a focus and definition of free and open source software. If you look at the Open Source Initiative, which has served to define what "open source" means, it is basically just a list of approved licenses. If you use one of those licenses, the software is open source, if not then it is closed.

I think this is disappointing, because licenses are just law, and law is not very interesting. A law tells you what you shouldn’t do, it doesn’t tell you what you should do. When both sides are talking about freedom, the licenses just define freedom as the lack of certain restrictions. Take away those restrictions and voila, you are free… as though we are all just bundles of freedom waiting to be released.

Self-Definitions

With licenses we have a negative definition of our community. Either license you choose, the license feels like a reaction against closed source software. If you can imagine a world in which there was no copyright, where our platforms were all setup to distribute source code in modifiable forms, where everything was open and everything was free, then none of these licenses would matter. No one would be compelled to create the GPL in such a world; we wouldn’t advocate for copyright just so we can secure people’s freedoms. In that kind of world all this licensing disappears. And this isn’t even so weird a world. You can pretend there’s no copyright now. Maybe you have to reverse-engineer some stuff. There’s lots of places in the world where no one really gives a damn about copyright. But those places don’t feel open source to me, they don’t feel that more free. We aren’t made unfree just by legal restrictions; freedom is something we have to actively grasp.

I don’t think what we do is predicated on copyright. Indeed, many projects are comfortable with an entirely confused copyright ownership. This causes very few problems. A focus on licensing makes us into a reaction against proprietary software, where we allow proprietary software and its makers to define what it means to be us.

This concerns me because it isn’t just about formal definitions and terminology. When I say what do I do, I say I am an open source programmer. That’s not just an attribute, like saying that my hair is brown. Open source is a way in which I see myself, a way I think about my craft, my profession, and a way I justify the work I put out to the world: that it aligns with these values. So it’s very important to me what these values are. And it’s frustrating to see open source defined in reaction to closed source software, because personally I don’t care about closed source software that much.

I never really cared much about fighting Microsoft, and I certainly don’t care now. I see myself as a builder; this is what always drew me towards programming. The desire to build new things. This is our privilege as programmers, that we always have the opportunity to build new things. If we’re asked to do something again and again and again, you always have the power to do it in a more abstract way, to generalize it away, until you can start to ignore the requests and move on to another problem. This is something unique thing to computer programming. These are the kind of unique attributes that make us different as a profession and as a craft than just about anything I can think of.

So I’m frustrated. Here we are stuck in this notion of a license as a self-definition. I want to find a new self-definition, a new idea of what makes us us.

What Makes Us Us

So… what makes us us?

I was saying about Django, the community is not particularly enthusiastic about philosophy. Or maybe I should say, Django’s philosophy is: the value of the code is the thing you do with it. These abstract discussions about architecture, reuse, big plans… instead, Django as a community encourages you to keep your head in the code, think about what you want to do, and then do it. Don’t shave yaks.

But I’m not here to tell you to get philosophical about freedom, or to berate you for a functional definition of value. I’d like to look at this community for what it is, and ask: what is the value system here? Maybe it isn’t described, but I also don’t think it is therefore absent.

So… when I say I identify as an open source programmer, what is it that I am identifying as?

I don’t believe licensing makes something truly open source. There was this clamour in the past to get companies to open source their products. This has stopped, because all the software that got open source sucked. It’s just not very interesting to have a closed source program get open sourced. It doesn’t help anyone, because the way closed source software is created in a very different way than open source software. The result is a software base that just does not engage people in a way to make it a valid piece of software for further development. At least not unless you have something peculiar going on… an economic force like you had behind Mozilla that could push things forward even in the midst of all the problems that project had. One might even ask, is Mozilla still suffering from that proprietary background, when something like KHTML or WebKit which came from a truly open source background, and has been a more successful basis for collaboration and new work.

So whatever it is that makes something open source, it’s not just licensing. Given a codebase, we can’t necessarily expect that someone going to care about it and love it and make it successful. A lot of people have described what makes a successful open source codebase; I’d like to talk some about what the communities look like.

Open source works as a fairly loose federation of people together. Everyone involved is involved as an individual. Companies seldom participate directly in open source. Companies might use open source, they might sponsor people to work on open source projects, they might ask an employee to act as a liason. But it’s not cool to submit a bug as a company. You submit it as yourself. If someone asks a question, you answer as yourself. You don’t join a mailing list under the company’s name. And even when you put a company name on a product, it’s hard to relate to the product as a project without some sense of authorship, of the underlying individual.

There’s also very little money being moved about. There’s not a lot of commercial contracts. You might get software, you might get bug fixes, you might get reputation, but there’s seldom any formal way setup to introduce commerce into the community. How many projects let you pay to escalate a bug? Even if everyone involved might like that, it’s just not there.

But I want to get back to individuals. How things are created is not that someone determines a set of priorities, lays them out, then people work on implementation based on those priorities. That of course is how things typically work at a company, as an employee. But open source software and open source projects are created because an individual looks at the world and sees an opportunity to create something they think should exist. Maybe it resolves a tension they’ve felt in their work, maybe it allows that person to respond to the priorities from above better, but the decision to implement lies primarily with the implementor. When someone makes a decision to move a product from simply private code — regardless of the license — to being a real open source project, that decision is almost always driven by the programmer.

Underneath most open source work there is a passion for the craft itself. This is what leads to a certain kind of quality that is not the norm in closed source software. It’s not necessarily less bugs or more features, but a pride in the expression itself. A sense of aesthetic that applies to even the individual lines of software, not just to the functionality produced. This kind of aesthetic defies scheduling and relies on personal motivation.

As open source programmers we are not first concerned with how a task fits into institutions, how a task can be directed by a hierarchy or an authority, or even how the task can be directed by economics. The tasks that we take on are motivated by aesthetic, by personal excitement and drive.

We are also in a profession where there is little stasis. If you can create something once, you can create it a thousand times, through iteration or abstraction. You can constantly make your own effort obsolete. A good programmer is always trying to work themselves out of a job.

Djangocon didn’t exist a couple years ago. Django didn’t exist only a few years ago. And I don’t think there’s anyone here who thinks that, having found Django, they’ve reached some terminal point. It’s hardly even a point to pause. There’s a constant churn, a constant push forward that we’re all participating in.

As a result it’s demanded of us that we have a tight feedback cycle, that education is not a formal affair but a constant process in our own work. There’s a constant churn, and a professional sense we’re kind of like fruit flies. A generation of knowledge and practice is short enough that the evolution is rapid and visible. You don’t have to be particularly old or even thoughtful to see the changes. You can look back even on your own work and on communities to see changes over the course of a couple years, to see changes and shifts and a maturing of the work and the community.

Another attribute of open source: our communities are ad hoc and temporary. We do not overvalue these communities and institutions; we regularly migrate, split, recombine, and we constantly rewrite. There is both an arrogance and a humility to this. We are arrogant to think This Time Will Be Different. But we are humble enough to know that last time wasn’t different either. There will always be a next thing, another technique, another vision.

Because of the ad hoc nature of the communities, we don’t have long collective plans. The ad hoc community may be the intersection of different personal long range plans, a time when different visions somehow coincide in a similar implementation. Or perhaps it’s just serendipity, or leadership. But we make each decision anew. I believe this protects us from being misled by sunk costs. The idea of a sunk cost is that when you make an investment, you’ve put in effort, that effort is gone. Just because you’ve put in effort doesn’t mean you’ve received value, or that the path of investment remains valid. But as humans we are highly reluctant to let go of a plan that we’ve invested in. We have a hard time recognizing sunk costs.

I believe in our developer community we approach our work with sufficient humility that we can see our work as simply a sunk cost. Our effort does not entitle us to any particular success, so we can choose new directions with more rationality than an institution. Though it can also be argued that we are too quick to dismiss past investments; there is a youthfulness even to our oldest members.

We do not have hierachies with decision makers above implementors. Some people have veto power (a BDFL), but no one has executive power. A decision only is valid paired with an implementation. You cannot decide something based on information you wish was true; you cannot decide on something then blame the implementors for getting it wrong. We are in this sense vertically integrated, decision and implementation are combined. The result may be success or failure, commitment or abandonment, but the hierarchy is flat and the feedback loop is very tight. And if an individual feels stymied, there is always another community to join or create.

Though this is only a start, it’s these attributes that I would prefer define us, not licenses.

I also would like that this could be a model for how other work should be done.

Why Us?

Why would we, as programmers, be unique or worthy of emulation? I mentioned before that we constantly work ourselves out of our job. We also create the tools we use to do the work. We define the structure of our communities. We’re consistently finding novel ways to use the internet build those communities. It’s not that we as a group are somehow uniquely wise, or some Greatest Generation, but we have become distinctly self-empowered. There is a uniqueness to us. It might be a coincidence of history, but it is there.

A question I might then ask: is there a political meaning to this? This is the form our craft takes, but does that mean anything? We work with computers, someone else might work with their hands, an artist considers color, a salesperson learns how to put on a good smile.

I haven’t quite figured this out yet, but I think there’s something in this. Over the years I’ve found myself looking at politics in a increasingly technocratic lens; more so than as a liberal, conservative, or radical. That is, instead of looking at the world and seeing what’s wrong about it, and explaining it in terms of a class struggle, a cultural conflict, in terms of advocacy or invented enemies or allies, I see a system that just works how it works. It’s more like gears than like a war. The gears turn and sometimes we don’t like what results, but it’s not malice.

But I also don’t think we are slaves to the technical functioning of the system. None of us are inevitably caught up in some statistical outcome of markets, or condemned by money in politics or advertising. At any time we can say Here Is What I Believe, and it is as powerful as any of those other things; we’re too quick to look at the people who aren’t asserting a belief, who aren’t asserting their own potential for self-empowerment and direction, and we ignore everyone who is aware and concerned and attempting self-determination. We are at danger of ignoring the great potential around us.

It is in this sense that I wonder not just how we can spread the idea of freedom through licensing, which has inspired the free culture movement, but also how we can spread this idea and practice of individual action, of combining decision and implementation, and of constant ad hoc collaboration.

I’m not just thinking of politics directly, but of professional lives as well. Right now we’re talking about healthcare. It’s a very political issue, and yet healthcare is ultimately a profession, a job, an action. How we work on that, collaboratively or not, is as political as any aspect of the system.

One anecdote that made me think about this, is a task I had that involved putting authentication in front of a mailing list. The mailing list happened to be for wound, ostomy, and continence nurses, and in the process of the job I read a bunch of their emails from the archives. As wound nurses they spent a lot of time asking about specific questions — maybe a wound that wouldn’t heal, they kept draining the puss and it discharge kept reappearing, and did anyone have ideas of the next technique to try?

Reading a few of these I could tell this was a profession where you needed a strong stomach. But the whole interaction, the way they described problems, the way people came back with answers, it felt very familiar to me. It was the same kind of discussions I could imagine having about Linux administration or debugging. And the goals were similar. No one was making money, there wasn’t really reputation on the line, it was just people who wanted to help their patients and who wanted to help each other.

So that mailing list was great, but it’s unfortunately not that common. And if nurses were open to that kind of collaboration, doctors don’t seem nearly as ready. And there’s a lot of professions where there’s not even that thoughtfulness. I believe in any profession there’s the ability to do it well or not; there’s nothing so rote or well understood that there’s no room for improvement. It doesn’t have to be fancy technology, it can just be a technique, a way of managing work; all things worth doing have some way of improving, by bringing in this same sense of collaboration and individual action and thoughtfulness, all things can be implemented better than they are now. What I’m describing isn’t a fancy new website for professionals, but about people look at their own work differently; the technology is not the hard part.

The Political

Changing how people look at their work I think is political. It involves individual empowerment. It can mean economic change. I also think it deemphasizes competition. When I think about Pylons, or Django, or TurboGears, or WSGI, there’s competition, but it’s also collegial. There’s not really that much of a sense of survival. We aren’t carving out territories, we’re just finding paths to some unknown end. If something else wins out, well, we’re all just along for the ride. In the end it is inevitable that something else other than what any of us are working on will win out over what any of us are doing. Just like everyone eventually loses their job at least to death or retirement. There’s no permanency. But if we can be individually more productive, it doesn’t have to mean we’ve put someone else out. It could mean we all, all of society, all of humanity, just do more. Why do we have to set ourselves against the Chinese, or Europe against the U.S.? Why do we have to set ourselves one economy against another?

Or consider government itself: we’re obsessed with our elected officials, but of course government is far larger than just the elected officials. The U.S. Federal Government alone has 1.8 million employees. We constantly threaten to institute accountability, meaning that we’ll poke and prod government workers from the outside and expect better outcomes. That we expect anything to come of this is absurd, but somehow accountability has become an easy alternative to constructive suggestions for improvement.

But why shouldn’t we expect that government workers want to do better? I believe in fact those people doing the work are especially well equiped to figure out how to do better. But it’s not automatic. They aren’t empowered in a system that is so exceptionally hierarchical. Lately we’ve seen lots of efforts to ask the public how to do government work better, but we’ve seen nothing asking government how to do government work better.

These are the kinds of things I’d like to see us all think about more: open source has done incredible things, has inspired new ideas, about more than just software and engineering, but I think we have yet more things to give.

Thank you for your time.

WebOb decorator

Ian Bicking — Sat, 23 May 2009 02:35:58 +0000

Lately I’ve been writing a few applications (e.g., PickyWiki and a revisiting a request-tracking application VaingloriousEye), and I usually use no framework at all. Pylons would be a natural choice, but given that I am comfortable with all the components, I find myself inclined to assemble the pieces myself.

In the process I keep writing bits of code to make WSGI applications from simple WebOb -based request/response cycles. The simplest form looks like this:

from webob import Request, Response, exc

def wsgiwrap(func):
def wsgi_app(environ, start_response):
req = Request(environ)
try:
resp = func(req)
except exc.HTTPException, e:
resp = e
return resp(environ, start_response)
return wsgi_app

@wsgiwrap
def hello_world(req):
return Response('Hi %s!' % (req.POST.get('name', 'You')))

But each time I’d write it, I change things slightly, implementing more or less features. For instance, handling methods, or coercing other responses, or handling middleware.

Having implemented several of these (and reading other people’s implementations) I decided I wanted WebOb to include a kind of reference implementation. But I don’t like to include anything in WebOb unless I’m sure I can get it right, so I’d really like feedback. (There’s been some less than positive feedback, but I trudge on.)

My implementation is in a WebOb branch, primarily in webob.dec (along with some doctests).

The most prominent way this is different from the example I gave is that it doesn’t change the function signature, instead it adds an attribute .wsgi_app which is WSGI application associated with the function. My goal with this is that the decorator isn’t intrusive. Here’s the case where I’ve been bothered:

class MyClass(object):
@wsgiwrap
def form(self, req):
return Response(form_html...)

@wsgiwrap
def form_post(self, req):
handle submission

OK, that’s fine, then I add validation:

@wsgiwrap
def form_post(self, req):
if req not valid:
return self.form
handle submission

This still works, because the decorator allows you to return any WSGI application, not just a WebOb Response object. But that’s not helpful, because I need errors…

@wsgiwrap
def form_post(self, req):
if req not valid:
return self.form(req, errors)
handle submission

That is, I want to have an option argument to the form method that passes in errors. But I can’t do this with the traditional wsgiwrap decorator, instead I have to refactor the code to have a third method that both form and form_post use. Of course, there’s more than one way to address this issue, but this is the technique I like.

The one other notable feature is that you can also make middleware:

@wsgify.middleware
def cap_middleware(req, app):
resp = app(req)
resp.body = resp.body.upper()
return resp

capped_app = cap_middleware(some_wsgi_app)

Otherwise, for some reason I’ve found myself putting an inordinate amount of time into __repr__. Why I’ve done this I cannot say.

Using pip Requirements

Ian Bicking — Wed, 17 Dec 2008 01:30:21 +0000

Following onto a set of recent posts (from James, me, then James again), Martijn Faassen wrote a description of Grok’s version management. Our ideas are pretty close, but he’s using buildout, and I’ll describe out to do the same things with pip.

Here’s a kind of development workflow that I think works well:

A framework release is prepared. Ideally there’s a buildbot that has been running (as Pylons has, for example), so the integration has been running for a while.
People make sure there are released versions of all the important components. If there are known conflicts between pieces, libraries and the framework update their install_requires in their setup.py files to make sure people don’t use conflicting pieces together.
Once everything has been released, there is a known set of packages that work together. Using a buildbot maybe future versions will also work together, but they won’t necessarily work together with applications built on the framework. And breakage can also occur regardless of a buildbot.
Also, people may have versions of libraries already installed, but just because they’ve installed something doesn’t mean they really mean to stay with an old version. While known conflicts have been noted, there’s going to be lots of unknown conflicts and future conflicts.
When starting development with a framework, the developer would like to start with some known-good set, which is a set that can be developed by the framework developers, or potentially by any person. For instance, if you extend a public framework with an internal framework (or even a public sub-framework like Pinax) then the known-good set will be developed by a different set of people.
As an application is developed, the developer will add on other libraries, or use some of their own libraries. Development will probably occur at the trunk/tip of several libraries as they are developed together.
A developer might upgrade the entire framework, or just upgrade one piece (for instance, to get a bug fix they are interested in, or follow a branch that has functionality they care about). The developer doesn’t necessarily have the same notion of "stable" and "released" as the core framework developers have.
At the time of deployment the developer wants to make sure all the pieces are deployed together as they’ve tested them, and how they know them to work. At any time, another developer may want to clone the same set of libraries.
After initial deployment, the developer may want to upgrade a single component, if only to test that an upgrade works, or if it resolves a bug. They may test out combinations only to throw them away, and they don’t want to bump versions of libraries in order to deploy new combinations.

This is the kind of development pattern that requirement files are meant to assist with. They can provide a known-good set of packages. Or they can provide a starting point for an active line of development. Or they can provide a historical record of how something was put together.

The easy way to start a requirement file for pip is just to put the packages you know you want to work with. For instance, we’ll call this project-start.txt:

Pylons
-e svn+http://mycompany/svn/MyApp/trunk#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk#egg=MyLibrary

You can plug away for a while, and maybe you decide you want to freeze the file. So you do:

$ pip freeze -r project-start.txt project-frozen.txt

By using -r project-start.txt you give pip freeze a template for it to start with. From that, you’ll get project-frozen.txt that will look like:

Pylons==0.9.7
-e svn+http://mycompany/svn/MyApp/trunk@1045#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk@1058#egg=MyLibrary

## The following requirements were added by pip --freeze:
Beaker==0.2.1
WebHelpers==0.9.1
nose==1.4
# Installing as editable to satisfy requirement INITools==0.2.1dev-r3488:
-e svn+http://svn.colorstudy.com/INITools/trunk@3488#egg=INITools-0.2.1dev_r3488

At that point you might decide that you don’t care about the nose version, or you might have installed something from trunk when you could have used the last release. So you go and adjust some things.

Martijn also asks: how do you have framework developers maintain one file, and then also have developers maintain their own lists for their projects?

You could start with a file like this for the framework itself. Pylons for instance could ship with something like this. To install Pylons you could then do:

$ pip -E MyProject install \\
> -r http://pylonshq.com/0.9.7-requirements.txt

You can also download that file yourself, add some comments, rename the file and add your project to it, and use that. When you freeze the order of the packages and any comments will be preserved, so you can keep track of what changed. Also it should be ameniable to source control, and diffs would be sensible.

You could also use indirection, creating a file like this for your project:

-r http://pylonshq.com/0.9.7-requirements.txt
-e svn+http://mycompany/svn/MyApp/trunk#egg=MyApp
-e svn+http://mycompany/svn/MyLibrary/trunk#egg=MyLibrary

That is, requirements files can refer to each other. So if you want to maintain your own requirements file alongside the development of an upstream requirements file, you could do that.

Inverted Partials

Ian Bicking — Tue, 09 Sep 2008 03:12:44 +0000

I was talking with a coworker some time ago about his project, and he needed to update a piece of the page in-place when you go back to the page, and setting the page as uncacheable didn’t really work. Which probably makes sense; I think at one time browsers did respect those cache controls, but as a result going back in history through a page could cause some intermediate page to be refreshed and needlessly slow down your progress.

Anyway, Rails uses partials to facilitate this kind of stuff in a general way. Bigger chunks of your page are defined in their own template, and instead of rendering the full page you can ask just for a chunk of the page. Then you do something like document.getElementById('some_block').innerHTML = req.responseText. Mike Bayer just described how to do this in Mako too, using template functions.

When asked, another technique also occurred to me, using just HTML. Just add a general way of fetching an element by ID. At any time you say "refresh the element with id X", and it asks the server for the current version of that element (using a query string variable document_id=X) and replaces the content of that element in the browser.

The client side looks like this (it would be much simpler if you used a Javascript library):

Then you need the server-side component. Here’s something written for Pylons (using lxml.html, and Pylons 0.9.7 which is configured to use WebOb):

from pylons import request, response
from lxml import html

def get_id(response, id):
if (response.content_type == 'text/html'
and response.status_int == 200):
doc = html.fromstring(response.body)
try:
el = doc.get_element_by_id(id)
except KeyError:
pass
else:
response.body = html.tostring(el)
return response

class BaseController(WSGIController):
def __after__(self):
id = req.GET.get('document_id')
if id:
get_id(response, id)

Though I’m not sure this is appropriate for middleware, you could do it as middleware too:

from webob import Request
class DocumentIdMiddleware(object):
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
req = Request(environ)
id = req.GET.get('document_id')
if not id:
return self.app(environ, start_response)
resp = req.get_response(self.app)
resp = get_id(resp, id)
return resp(environ, start_response)

My Experience Writing a Build System

Ian Bicking — Fri, 20 Jun 2008 04:41:54 +0000

Lately there’s been some interest in build processes among various people — Vellum was announced a while back, Ben has been looking for a tool and looking at Fabric, and Kevin announced Paver. At the same time zc.buildout is starting to gain some users outside of the Zope world, and I noticed Minitage as an abstraction on top of zc.buildout.

A while ago I started working on a build project for Open Plans called fassembler. I think the result has been fairly successful and maintainable, and I thought I’d share some of my own reflections on that tool.

Update: what we were trying to accomplish

I didn’t make it clear in the post just what we were trying to do, and what this build system would accomplish.

Our site (openplans.org) is made up of several separate servers with an HTML-rewriting proxy on the front end. We have a Zope server running a custom application, Apache running WordPress MU, and some servers running Pylons or other Python web applications for portions of our site. We needed a way to consistently reproduce this entire stack, all the pieces, plugged together so that the site would actually work. Two equally important places where we had to reproduce the stack are for developer rigs and the production site.

Our code is primarily Python and we use a lot of libraries, developed both internally and externally. Setting up the site is primarily a matter of installing the right libraries and configuration and setting up any databases (both a ZODB databases and several MySQL databases). We use a few libraries written in C, but distutils handles the compilation of those pretty transparently.

For this case we really don’t care about build tools that focus on compilation. We don’t care about careful dependency tracking because we are compiling very little software.

make doesn’t make sense

Update 2: If you think the make model makes lots of sense, read the preceding section — it makes sense for a different problem set than what we’re doing.

We initially had a system based on BuildIt, which is kind of like make with Python as the control code. It wasn’t really a good basis for our build tool, and I think it added a lot of confusion, compounded by the fact that we weren’t quite sure what we wanted our build to do. Ultimately I think the make model of building doesn’t make sense.

The make model is based on the idea that you really want to save work. So you detect changes and remake things only as necessary. For compilation this might make sense, because you edit code and recompile a lot and it’s tedious to wait. But we are building a website, and installing software, and none of that style of efficiency matters. make-style detection of work to be done doesn’t even save any time. But it does make the build more fragile (e.g., if you define a dependency incorrectly) and much harder to understand, and you constantly find yourself wiping the build and starting from scratch because you don’t trust the system.

The metaphor for the new build system was much simpler: do a list of things, top to bottom. There’s no effort into detecting changes in the build, or changes in the settings, or anything else.

Do things carefully

In the build system almost all actions go through the filemaker module. This is kind of a file abstraction library. But the goals are entirely different than convenience: the goal is transparency and safety. In contrast Paver uses path.py for convenience, but I’m not sure what the win would be if we used a model like that.

filemaker itself is heavily tied to the framework that it’s written for, specifically user interaction and logging. Most tasks just do things, and rely on filemaker to detect problems and ask the user questions. For example, every time a file is written, it checks if the file exists, and if it has the same content. If it exists with other content, it asks the user about what to do. It doesn’t overwrites files without asking (at least by default). I think this makes the tool more humane as the default behavior for a build is to be careful and transparent. The build author has to go out of their way to make things difficult.

Many zc.buildout recipes will blithely overwrite all sorts of files which always made me very uncomfortable with the product. It’s the recipes in zc.buildout which do this, not the buildout framework itself, but because buildout made overwriting the easy thing to do, and didn’t start with humane conventions or tools, this behavior is the norm.

What I think filemaker most accomplished was the ability to do file operations while also asserting the expected state of the system, and so makes build bugs noticeable earlier instead of getting a build process that finishes successfully but creates a buggy build, or having an exception show up far from where the error was originally introduced.

Also, because it won’t overwrite your work in progress this has saved the build from engendering deep feelings of hatred in cases when it might overwrite your work in progress. It’s hard to detect this absence of hatred, but I know that I’ve felt it with other systems.

Update: a corollary: ignore no errors

One question you might wonder about: why not a shell script? We did prototype some things as shell scripts, but we’ve consistently moved to Python at some point, even things that seemed really trivial. The problem with shell scripts is they have horribly bad behavior with respect to errors. Ignoring errors is really really easy, noticing errors is really hard.

This is absolutely unacceptable for builds. Builds must not ignore errors. The build may mostly work despite an error. It might be totally broken, but the error message is lost in all sorts of useless output. The error message probably makes no sense. The context is lost. No suggestion is given to the user.

When builds work, that’s great. Build do not always work. They always fail sometimes, and some poor sucker (usually in some hot potato-like arrangement) has to figure out what went wrong. You have to plan for these problems.

Everything in the build tries to be careful about errors. All places where it is not, it is a bug. The resolution isn’t to see something appear to work, but create a broken build, and say "oh, you forgot to set X". The resolution is to make sure when you forget to set X it gives you an error that tells you to set X.

This is one of the more important and more often ignored principles of a good build/deployment system. Maybe it’s gotten better, but when I first used zc.buildout (very early in its development) the poor handling of errors was by far the biggest problem and it left me with a bad taste in my mouth. easy_install and setuptools in general is also very flawed in this respect.

Log interesting things

I tried to make a compromise between logging very verbosely, and being too quiet. As a user, I want to see everything interesting and leave out everything boring. Determining interesting and boring can be a bit difficult, but really just require some attention and tweaking.

To make it possible to visually parse the output of the tool I found both indentation and color to be very useful. Indentation is used to represent subtasks, and color to make sections and warnings stand out.

The default verbosity setting is not to be completely quiet. Silence is a Unix convention that just doesn’t work for build tools. Silence gets you interactions like this:

$ build-something target-directory/
(much time passes)
Error: cannot write /home/ianb/builds/20080426/target-directory/products/AuxInput/auxinput/config/configuration.xml

Why did it want to write that file? Why can’t it write that file? Is the build buggy? Did I misconfigure it? Does the directory exist?

The typical way of handling this is either to run the build again with logging setup or otherwise make it more verbose, or to get in the habit of always running it verbose.

Mixing code and configuration

BuildIt, which we were using before, had the ability to put variables in settings, and you could read an option from another section with something like ${section/option}. It was limited to simple (but recursive) variable substitution, and had some clever but very confusing rules that created a kind of inheritance.

I liked the ability to do substitution, but wasn’t happy with the compromise BuildIt made. I wasted a lot of time trying to figure out the context of substitutions. So, I saw two directions. One was to remove the cleverness and just do simple substitution. This is the choice zc.buildout made. The other was to go whole-hog. With a bit of trepidation I decided to to go for it, and I made the choice to treat all configuration settings as Tempita templates. All configuration is generally accessed via config.setting_name, and that lazily interpolates the setting (it took me quite a while to figure out how to avoid infinite loops of substitution). Because evaluation is done lazily settings can depend on each other and be overridden and have lots of code in defaults (e.g., a default that is calculated based on the value of another setting), and it works out okay. Most settings just ended up having a smart default, and as a result very little tweaking of the configuration is necessary.

Somewhat ironically the result was a kind of atrophying of the settings, because no one actually set them, instead we just tweaked the defaults to get it right. Now I’m not entirely sure what exactly the "settings" are setting, or who they should really belong to. To the build? To the tasks? While this is conceptually confusing, in practice it isn’t so bad. This mixing of code and configuration has been distinctly useful, and not nearly as problematic to debug as I worried it would be. In some ways it was a way of building lambda into every string, and the lazy evaluation of those strings has been really important. But it’s not clear if they are really settings.

Would normal string interpolation have been enough (e.g., with string.Template)? I’m pretty sure it wouldn’t have been. The ability to do a little math or use functions that read things from the environment has been very important.

Managing Python libraries

fassembler uses virtualenv for building each piece of the stack. Generally it creates several environments and installs things into them — it doesn’t run inside the environments itself. This works fine.

zc.buildout in comparison does some fancy stuff to scripts where specific eggs are enabled when you run a script. Each script has a list of all the eggs to enable. You can’t install things or manage anything manually, even to test — you always have to go through buildout, and it will regenerate the scripts for you. zc.buildout was implemented at the same time as workingenv (the predecessor to virtualenv), and I actually finished virtualenv with fassembler in mind, so I can’t blame zc.buildout for not using virtualenv. That said, I don’t think the zc.buildout system makes any sense. And it’s really complicated and has to access all sorts of not-really-public parts of easy_install to work.

Isolation is only the start. easy_install makes sure each library’s claimed dependencies are satisfied. You might then think easy_install would do all the work to make the stack work. It is nowhere close to making the stack work. setup.py files can/should contain the bare minimum that is known to be necessary to make a package work. But they can’t predict future incompatibilities, and they can’t predict interactions. And you don’t want all your packages changing versions arbitrarily. If you work with a lot of libraries you need those libraries to be pinned, and only update them when you want to update them, not just because an update has been released.

So for each piece of the stack we have a set of "requirements". This is a flat files that indicates all the packages to install. They can have explicit versions, far more restrictive than anything you should put in setup.py. It also can check out from svn, including pinning to revisions. This installation plan can go in svn, you can do diffs on it, you can branch and copy and do whatever. Maybe at some point we could use it to keep cached copies of the libraries. For now it mostly uses easy_install (and python setup.py develop for checkouts).

In parallel we have a command-line program for just installing packages using files like this, called PoachEggs. I want to make this better, and have fassembler use it, but I mostly note it because it implements a feature that can "freeze" all your packages to a requirements file. You take a working build and freeze its requirements, giving explicit (==) versions for packages, and pin all the svn checkouts to a revision, so that the frozen requirements file will install exactly the packages you know work.

An alternative to this is what the Repoze guys are doing, which is to create a custom index that only includes the versions of libraries that you want. You then tell easy_install to use this instead of PyPI. It works with zc.buildout (and anything that uses easy_install), but I can’t get excited about it compared to a simple text file. I also want svn checkouts instead of create tarballs of the checkout — I like an editable environment, because the build is just as much to support developers as to support deployment.

The structure

A big part of the development of fassembler was nailing down the structure of our site, and moving to use tools like supervisor to manage our processes. A lot of these expectations are built into the builds and fassembler itself. This is part of what makes the build Work — the pieces all conform to a common structure with some basic standards. But this isn’t the build tool itself, it’s just a set of conventions.

I don’t know quite what to make of this. Extracting the conventions from the builds leads to a situation where you can more easily misconfigure things, and the installation process ends up being more documentation-based instead of code-based. We do not want to rely on documentation, because documentation is generally because of a flaw in the build process that needs explaining. It’s faster for everyone if the code is just right. Maybe these conventions could be put into code, separate from the build. The abstraction worries me, though — too much to keep track of?

What we don’t get right

The biggest problem is that fassembler is our own system and no one else uses it. If someone wants to use just a piece of our stack they either have to build it manually or they have to use our system which is meant to build all our pieces together with our conventions. There’s some pressure to use zc.buildout to make pieces more accessible to other Zope users. We’ve also found things that build with zc.buildout that we’d like to use (e.g., setups for varnish).

We haven’t figured out how to separate the code for building our stuff from the build software itself. There’s a bootstrapping problem: you need to get the build code to build a project, and so it can’t be part of the project you are building. zc.buildout uses configuration files (that aren’t code, so they lack the bootstrap problem) and it uses recipes (a kind of plugin) and has gone to quite a bit of effort to bootstrap everything. virtualenv also supports a kind of bootstrap which we use to do the initial setup of the environment, but it doesn’t support code organization in the style of zc.buildout.

Builds are also fairly tedious to write. They aren’t horrible, but they feel much longer than they should be. Part of their length, though, is that over time we put in more code to guard against environment differences or build errors, and more code to detect the environment. But compared to zc.buildout’s configuration files, it doesn’t feel quite as nice, and if it’s not as nice sometimes people are lazy and do ad hoc setups.

The future

We haven’t really decided, but as you might have noticed zc.buildout gets a lot of attention here. There’s quite a few things I don’t like about it, but a lot of these have to do with the recipes available. We don’t have to use the standard zc.buildout egg installation recipe. In fact that would be first on the chopping block, replaced with something much simpler that assumes you are running inside a virtualenv environment, and probably something that uses requirement files.

Also, we could extract filemaker into a library and recipes could use that. Possibly logging could be handled the same way (the logging module just isn’t designed for an interactive experience like a build tool). Then if we used other people’s recipes we might feel grumpy, since they’d use neither filemaker or our logging, but it would still work. And our recipes would be full of awesome. The one thing I don’t think we could do is introduce the template-based configuration. Or, if we did, it would be hard.

That said, there is a very different direction we could go, one inspired more by App Engine. In that model we build files under a directory, and that directory is the build. Wherever you build, you get the same files, period. All paths would be relative. All environmental detection would happen in code at runtime. Things that aren’t "files" exactly would simply be standard scripts. E.g., database setup would not be done by the build, but would be a script put in a standard location.

This second file-based model of building is very much different than the principles behind zc.buildout. zc.buildout requires rebuilding when anything changes, and does so without apology. It requires rebuilding to move the directories, or to move to different machines. Using a file-based model requires a lot of push-back into the products themselves. Applications have to be patched to accept smart relative paths. They have to manage themselves a lot more, detect their environment, handle any conflicts or ambiguities, being graceful about stuff like databases, because the files have to be universal. In an extreme case I could imagine going so far as to only keep a template for a configuration file, and write the real configuration file to a temporary location before starting a server (if the server cannot be patched to accept runtime location information).

So this is the choice ahead. I’m not sure when we’ll make this choice (if ever!) — build systems are dull and somewhat annoying, but they are no more dull and annoying than dealing with a poor build system. Actually, they are definitely less dull than working with a build system that isn’t good enough or powerful enough, or one that simply lacks the TLC necessary to keep builds working. So no choice is a choice too, and maybe a bad choice.

Governance

Ian Bicking — Tue, 06 May 2008 04:08:30 +0000

It occurred to me… Django is something like a dictatorship… or maybe an oligarchy. At first it seems like Pylons is the same… but no. Pylons is clearly feudal. I lord over Paste, WebOb, FormEncode. Mike Bayer lords over Mako and SQLAlchemy. Ben lords over Routes, Beaker, and Pylons.

I suppose in all cases there is a certain amount of democracy, because there are no serfs, and any individual is free to travel to any kingdom they like. Well, at least among the open source kingdoms. Without citizenship, and with no exclusiveness of ownership, with even property having largely disappeared, I suppose it’s inevitable that traditional metaphors of control and governance don’t really make sense.

App Engine and Pylons

Ian Bicking — Sun, 13 Apr 2008 17:32:06 +0000

So I promised some more technical discussion of App Engine than my last two posts. Here it is:

Google App Engine uses a somewhat CGI-like model. That is, a script is run, and it uses stdin/stdout/environ to handle the requests. To avoid the overhead of CGI a process can be reused by defining __main__.main(). But while a process can be reused, it might not be, and of course it might get run on an entirely separate server. So in many ways it’s like the CGI model, with a small optimization so that, particularly under load, your requests can run with less latency.

This part is all well and good. I’ve already come to terms with servers going up and down without warning. But the environment itself has a number of other restrictions. It seems that App Engine is providing security in the language itself. The interpreter has been modified so that code is sandboxed, with no ability to write to the disk, open sockets, import C extensions, and see quite a few things in its environment. It’s these things that are a bit harder to come to terms with.

While they claim it supports any Python framework, these restrictions don’t actually make it easy. So for the last few days quite a few of us have been hacking various things to get stuff working.

The first thing people noticed is that Mako and Genshi didn’t work, because they use the ast (via the parser module) to handle the templating, and that module has been restricted. Apparently arbitrary bytecode is not safe in this environment, and so anything that can produce bytecode is considered dangerous. From what I understand Philip Jenvy has been working on Mako and the trunk is currently working. He’d already been doing work to get Mako working on Jython, which had similar issues. Genshi is also in progress and fairly close to working, though with some missing features. Genshi has the harder task as Mako was primarily reading the ast, while Genshi was writing it.

The first thing I noticed is that Setuptools doesn’t work. I’m flattered that one of the only 3 libraries included with App Engine is WebOb, but of course I am more enamored of a rich set of reusable libraries. Setuptools didn’t work because several modules and functions have been removed — this like os.open, os.uname, imp.acquire_lock, etc. Some of these are kind of reasonable, while others are not. The removal of many functions from imp doesn’t really make sense, for instance (I think the motivation was the difficulty of auditing the implementation of those functions, not that the functionality itself is dangerous). And while some functions can’t be used in the environment, the fact you can’t import those functions is more problematic. For instance, The Setuptools’ pkg_resources module has support to unzip eggs when they are imported. App Engine doesn’t support importing from zip files at all, and you certainly can’t unzip to a temporary location. But withoutthe necessary modules and objects pkg_resources won’t even import.

To work around this I started a new project: appengine-monkey, which adds several monkeypatches and replacement dummy modules to the environment to simulate a more typical environment. It’s just a small list so far (mostly in this module), but I expect as people experiment with other libraries the list will increase. For example, I would welcome implementations of things like httplib on top of urlfetch in this library. (Implementing httplib and stubbing out parts of socket would probably make urllib run.)

But the good news is that Pylons is pretty much working on App Engine, as is Setuptools and you can manage your libraries using virtualenv.

The instructions are all located in the appengine-monkey Pylons wiki page. Please leave comments if you have improvements or problems with that process. I also welcome contributors and developers to the project itself — this is a project for expediting App Engine development, it is not a project I care to champion or control. Or support to any large degree.

One ticket which is rather important is the apparent maximum number of files and blobs: 1000. Libraries involve lots of files, and the base Pylons install is only barely under this limit. Now I just wish I could use lxml, but that’s probably going to be a long time coming.

Update: As of April 2009 these issues were fixed; it took a year, but at least it’s done. The 1000 file limit has been relaxed (1000 code plus 1000 static) but still exists. lxml remains unlikely.

App Engine and Open Source

Ian Bicking — Wed, 09 Apr 2008 19:13:36 +0000

This is about Google App Engine which probably everyone has read about already.

I’m quite excited about it. Hosting has been the bane of the Python web world for a long time. This provides a very compelling hosting situation for Python applications.

I’m not as interested in this from a competitive perspective as I am from a simple this-is-awesome perspective. Regardless of how this positions Python relative to other languages, this is something Python needs. But even looking beyond that, I think this is something the open source world needs. Open source web development is in a funny place. There’s a lot of reasons why web programming is a good domain for open source. The barrier to entry for web development is extremely low. Developers have choice in their tools, as browsers don’t really care what software you use so long as you serve up HTML. This leads to experimentation and excitement and the kind of self-direction that is very motivating to developers. It leads to the kind of personal excitement that underlies most open source development.

Despite this, open source web application development doesn’t seem sustainable. There’s some applications, sure. WordPress, Trac, MediaWiki, MoinMoin. But most wiki software doesn’t have a vibrant community. Many a bug tracker has fallen by the wayside. Blog software projects have a horrible time building a viable community. Other website software hardly gets anywhere at all. A lot of the development that might appear to be application development really is more like a framework when you look closely (e.g., Plone, Drupal).

I think deployment concerns are a huge part of this. And, given its better deployment story, it’s no surprise PHP is the basis of most viable open source web applications. Being interested in a project requires that you be able to use the product (and usually use it casually, as that’s the point of entry for many developers). Right now most people can’t use open source web applications.

But people can use hosted applications, and that’s where all the effort has gone in the past few years. I am comfortable saying that Trac is a better issue tracker than Google Code’s issue tracker. But I’d probably recommend Google Code to someone starting a new project, because it’s so much less work. Similarly I’d try to dissuade most people from installing their own blog software. I still don’t know what to tell people about a CMS.

Many people are excited about how far up you might be able to scale something based on App Engine, but (like Dave) I’m excited about how far it could be scaled down. For the majority of sites the free quota will be more than enough. But that alone isn’t the point: there’s lots of free services people can use. The difference here is that the free services can be modified and controlled by the anyone who signs up and installs an application.

From the perspective of open source it’s a bit awkward that the platform itself is proprietary. Questions about sharecropping are a valid concern, but I’m optimistic about the ultimate outcome. The SDK is under a permissive open source license, and the APIs are all reasonable enough that they could be reimplemented with open source backends (maybe without the same scalability, but that’s not the aspect I care about anyway). Perhaps the BigTable APIs will serve as the basis for future storage APIs.

But even if other people make compatible implementations of these APIs, would it matter? If Google offers free hosting, is someone else really going to be able to provide a better hosting option? Or would these other implementations just be strawmen, a way to show that It Could Be Done? If the libraries are just written to prove a point, I can’t see them gaining much traction. But I think these could be viable as there are other constraints to the App Engine environment that people may want to escape at some point in their application development.

As to the details of App Engine? Can you run Pylons or Paste on it? Well, that’s a topic for another post.

Update: I wrote up some more thoughts