Lately there’s been some interest in build processes among various people — Vellum was announced a while back, Ben has been looking for a tool and looking at Fabric, and Kevin announced Paver. At the same time zc.buildout is starting to gain some users outside of the Zope world, and I noticed Minitage as an abstraction on top of zc.buildout.

A while ago I started working on a build project for Open Plans called fassembler. I think the result has been fairly successful and maintainable, and I thought I’d share some of my own reflections on that tool.

$ build-something target-directory/
(much time passes)
Error: cannot write /home/ianb/builds/20080426/target-directory/products/AuxInput/auxinput/config/configuration.xml

I should preface this entire post by noting that I haven’t used Erlang, just read about it, and I handle most concurrency using Python threads, a model for which I have no great affection (or hate). But I was reading two posts by Patrick Logan on Erlang and it got me thinking again.

From my reading and what I’ve heard from other people, Erlang the language-and-syntax seems quite crufty. Erlang is not the Complete Package, the language that will make you leave your wife, the language that will have you walking into telephone polls because it’s just so hot that you can’t take your eyes off it. Erlang is not that language. People seem to get excited about two things in Erlang: the concurrency model and pattern matching. I’m not quite sure why people are excited about pattern matching. Pattern matching isn’t particularly hard to implement, though maybe it interacts with other aspects of the system in a way I don’t understand. I remain skeptical that it’s anything special.

Then there is the concurrency model (and its associated message passing), and this does strike me as special. You can kind of implement that model in Python, but with none of the concrete benefits.

These are the useful features I see in the concurrency model:

1. Erlang processes are share-nothing.
2. The processes are light weight. You can start lots of processes, and they start quickly and can die off quickly without any great overhead. They aren’t OS-level processes. These are sometimes called green processes or microprocesses. There are many implementations of microthreads in Python, but you lose the benefits of share-nothing.
3. The runtime makes light weight processes feasible as well. The OS overhead of a Python process is actually just a small part of the total overhead when spawning a process. Loading and initializing the runtime libraries of a Python program of even modest complexity is problematic. I’m not sure exactly how Erlang does this, but I suspect it’s because libraries can be safely shared because Erlang is a functional language. (PHP is also like this, with most of the library written in a sharable C library.)

1 and 2 can be handled with the right VM. This isn’t a particularly common VM design, but it’s certainly doable. 3 is tricky, and effects the language design.

I don’t think it effects the language design as greatly as to require a functional language. It requires that sharable code itself be a functional (i.e., immutable) subset of the language. What the code does doesn’t have to be functional, only what the code is.

To explain this, functions in most languages are immutable. Python functions aren’t actually immutable, but they are close enough that hardly anyone would notice if you made them immutable (right now you can overwrite their name, compiled code, and some other stuff — but almost no one actually does this). The function may have side effects, but as long as the function object is immutable then it can be shared safely among processes.

You can extend this to the module as a whole. This means you couldn’t monkeypatch objects in the module, and module-level assignments would effectively be constants. Any module-level objects would have to be immutable. Things like classes, which are also a kind of module-level object, would also have to be immutable — meaning things like class variables would be immutable. It could still be possible to do module-level code if there was a way of freezing the module after its instantiation (this would be important for Python, as even decorators are a kind of code run at import time). Adding a general concept of "freezing" to the language might be the most general and expedient way to make modules sharable.

The other half of the concurrency model in Erlang is message passing. Erlang processes send messages around like other systems send methods. It would be possible to use exactly Erlang’s system, as it’s not Erlang-specific and has been ported to many languages. Though I’d be somewhat more inclined to use bencode as it has some cleverness in its design. Or perhaps JSON, just because. Regardless of the format you are always passing around data, which I think is important. Systems that pass around "objects" become complex, mostly because you just cannot pass objects around and so those systems are just complex facades built around data exchange.

Lately I’ve become fond of thinking of objects as views over a more fundamental data structure (WebOb is strictly based on this pattern). Methods are details of the implementation, but only the data can mean something to someone else. At least this is the worldview you start to internalize when you think about REST a lot. (My post Documents vs. Objects is also about this.)

Arguable a plain-data-with-views world is just the dynamic-weak-typed anti-pattern. That anti-pattern is one where you get all the disadvantages of dynamic typing, and all the disadvantages of static typing, all in one ugly bundle. Erlang’s records remind me of this anti-pattern. Instead of dictionaries everything is a tuple, and getting a record is just syntactic sugar mapping record names to indexes — all in the context of a dynamically typed language where you could mistype a value and get strange output as a result.

The strong-static typing solution to the type problem involves complicated contracts, aka "service descriptions", the stuff of WS-*, CORBA IDL, etc. The strong-dynamic typing solution is that every object have an explicit type. This is fine for things we all agree on: lists, dictionaries, bytes, numbers. Sadly even a basic thing like unicode strings cause problems, as do dates, but at least those have straight-forward solutions (you add more basic types to the message format). More complex types, like a domain object (e.g., a user record) are difficult. Are they all just dictionaries? Dictionaries plus special metadata? XML namespaces address this problem in some ways, but are also the point at which XML starts to just piss people off and make them want to use JSON. And for good reason, because XML namespaces force you to define entities and responsible parties very early on, possibly long before you know what you are actually trying to do. Besides XML, are there other systems that aren’t just naive/unextendable (bencode, JSON) and are still basically dynamically typed?

This is the kind of thing I would really love to see PyPy experiment with.

Do you believe the world is (a) getting better, or (b) getting worse?

Please explain. Please, no more "both/neither" answers: choose just one

People have asked me a few times about evalexception and pdb — they’d like to be able to use something like pdb through the browser, stepping through code.

The technique I used for tracebacks wouldn’t really work for pdb. For a traceback I saved all the information from the frames — mostly just the local variables — and then let the user interact with that through the browser. But with pdb you pause the application part way through waiting for user input, and the routine only completes much later.

While writing WaitForIt I played around with techniques to deal with very slow WSGI applications. Not that hard, really — you launch every request in a new thread, and you manage those requests in an application of its own. So I started thinking about pdb again, and it started seeming feasible. Whenever the app reads from stdin it goes into an interactive mode, showing you what comes out on stdout and letting you add input to stdin. It’s nothing specific to pdb really.

So, with a bit of hacking, I added it into WebError (which is an extraction of the exception handling in Paste). To give the demo a try, do:

hg clone http://knowledgetap.com/hg/weberror/
cd weberror
python setup.py develop
# You need Paste trunk:
easy_install Paste==dev
python weberror/pdbcapture.py

What you’ll see is not polished, it’s just working, but since I mostly did it to see if I could do it, that’s good enough for me.

For the most part by the time I finished writing my last article on licensing I had mostly convinced myself that the GPL isn’t a practical license for most projects. That is, outcomes when using the GPL aren’t likely to be any better than outcomes using a permissive license, except for certain kinds of projects, mostly projects involving big faceless companies, and I’d just as soon avoid such projects anyway.

My own thinking on this has changed over the years in part because of a greater sense of humility about what I produce. I’m really not that worried about people stealing my work because I don’t think that theft would be of much value. But also because I realize that the value in software is not so much in the code as in the process. The process is what is valuable, particularly for open source, and licensing doesn’t really address issues of process.

As an example, if I’m uncomfortable with how some member of an open source community is using the code, or the community, I will be much more effective by dealing with that head-on, talking with that member, or even confronting them if it’s really necessary. If you give someone an unwelcoming attitude, they’ll probably go away. The license doesn’t need to be your gatekeeper. It’s not a particularly effective gatekeeper anyway.

Another change is perhaps a more reasonable valuation of code. There was a time when people wanted to protect their intellectual property. Even some non-software company might have gotten the idea that it should own the code it contracts someone else to write, under a proprietary license, so they could sell that software later. That anyone would care to buy it was always an illusion, but the illusion is a little more obvious these days.

One value of the GPL that I do want to acknowledge is its expression of values. It makes this explicit:

When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things.

To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others.

But the GPL does more than just its text: adopting the GPL is a statement of principle on the part of the original authors, of the people who adopt the project, and of the people who later help maintain the project. It is a statement that freedom is valued and that it is valued in a universal sense, not just in a personal or isolated sense.

This is implicit, not explicit, in the choice of license, but despite that I see this pattern in projects. Projects that choose the GPL are more likely to engender a spirit of openness and sharing. Not of the core project itself — both GPL and permissively licensed projects accomplish this just fine so long as they are properly maintained, and their success is far more related to how the project is managed than the licensing. But I see the difference in the sofware that grows up around the project: extensions, complementary projects, documentation.

Maybe this is because of licensing. The license filters the community, and the people who are left in a GPL project are all at least open to sharing. But more than that, I think it puts people in the right state of mind to share. The project feels more principled, the participation is based less on pragmatism and more on optimism. And there’s always people coming into open source who haven’t really figured out why or what they want to get out of it. Presenting them with the principles of Free Software influences their decision. (This issue has caused some debate about terminology.)

With all that said, you don’t need the GPL to present the principles of a project. It’s certainly the easiest way to do so. The GPL is shorthand for a rich set of principles and ideals. But it’s shorthand for people who are already in the know. The ideas need to be reiterated and explained and reconsidered to stay relevant. I think a project might do more good with an explicit statement of principles. With that in place the licensing might not matter so much.

It occurred to me… Django is something like a dictatorship… or maybe an oligarchy. At first it seems like Pylons is the same… but no. Pylons is clearly feudal. I lord over Paste, WebOb, FormEncode. Mike Bayer lords over Mako and SQLAlchemy. Ben lords over Routes, Beaker, and Pylons.

I suppose in all cases there is a certain amount of democracy, because there are no serfs, and any individual is free to travel to any kingdom they like. Well, at least among the open source kingdoms. Without citizenship, and with no exclusiveness of ownership, with even property having largely disappeared, I suppose it’s inevitable that traditional metaphors of control and governance don’t really make sense.

I thought I’d take some time to talk about licensing.

Licensing is something that F/OSS programmers talk about a lot. There’s two major categories of licenses:

• The GPL, aka Copyleft. You must distribute source with your application, and users get full rights to the source code, including any additions you may make.
• Permissive licenses, X/MIT, BSD, etc. These let you do pretty much everything.

There’s also the LGPL, which vaguely applies GPL-like terms to the original code, but is not "viral". LGPL originally meant "Library GPL" but was renamed to "Lesser GPL" because people automatically used it for libraries without considering what the license actually enforced.

How much do these licenses matter? How should a person choose between these options?

First, the LGPL. It has specific phrasing that makes some sense for C code, to distinguish between extending the library itself and using the library. It doesn’t make much sense for other runtime environments. Some people have tried to clarify its meaning in other environments, but I’m not sure if a clarification like that means much. The ambiguity seems like the worst of all worlds. People who are afraid of the GPL won’t use the software anyway. People who act in bad faith can satisfy the terms of the license through trickery. You don’t get any practical protection over a permissive license, and you get most of the stigma of the GPL. The GPL has some success stories, places where source was actually released due to the terms of the license. I don’t know of any similar success stories for the LGPL. If you don’t want to use the GPL, just use a permissive license.

Then the question: GPL or permissive licensing?

For some time I’ve chosen a permissive license for my code. But I’m not really advocating that, and every so often I throw out a little GPL code. Underneath this is something of a disinterest in licensing. I don’t think it means much. If specifics of your license matters then something has gone wrong. People are leeching code, and/or the community isn’t providing enough benefit to encourage participation. I don’t believe that software has much intrinsic worth, and treating it like property doesn’t make that much sense to me. Licensing treats software as property, which is why I see the relevance of licensing as a kind of disfunction. But there is the outside chance that it’s really just a big project, or that the project is being participated in by rivals. But nothing I work with is like this, and it’s pretty uncommon generally, and anyway those projects all have their licensing pretty much figured out.

The GPL does seem to serve a constructive purpose when rivals have to cooperate in some fashion. It makes a kind of demilitarized zone where everyone has to work for the collective good. But even this is a sign that you can’t trust the participants. This is somewhat reasonable, because you can’t actually trust large corporations with faceless programmers working on the code. But even that’s a kind of disfunction, because you can at least kind of trust large corporations with programmer employees who have a public face. The individual programmers are going to be very uncomfortable participating in any kind of Machiavellian conspiracy. Good open source projects are a coming together of individuals. Institutions are not effective participants. There are however scaling issues with individual participation: particularly that only a minority of developers are actually inclined to participate in this way. An institution can pay someone to work on something they don’t care very deeply about, and that person can still do useful work. But they won’t be part of the community.

Outside of this mediation of rivalries I’m not really sure what the GPL provides. It protects users, because the terms of the GPL ensure that users get code. It doesn’t actually ensure that code is made public, though it does give developer employees leverage to make their work public. There’s some urban legends about the viral aspect of the GPL, but that’s really just bullshit. Code doesn’t magically get relicensed just because there’s a little GPL in the mix. The GPL police won’t show up at your door and confiscate your computers. The only people who make those claims are GPL-haters like Microsoft. Who should you trust: the FSF or Microsoft? If you answered "Microsoft" then you need to get your head screwed on straight.

Unfortunately there is a stigma about the GPL, and there are competent developers that avoid all GPL software. I find this frustrating, especially since I don’t actually care about licensing, and so this adds a point of contention that I don’t really care to pay attention to. Some people in this situation then become GPL haters, because they feel like it’s all GPL’s fault. But the GPL didn’t create the licensing situation — proprietary software started this. Direct your hate where it belongs. The GPL ain’t it.

If my libraries were GPL I doubt I could leverage that to make other people open source their code. But I know I would alienate some people, and as a result I choose a permissive license because it’s just strategically advantageous.

Applications I think are different strategically. You don’t just swap an application in and out like you might a library. If you choose to use an application, then often the licensing is a secondary choice. The licensing ultimately only really applies to extensions. Libraries also have a different pattern of acceptance. Stubborn GPL haters can in effect veto a library in many projects (this is because stubborn GPL haters are viral). But in an application their influence is much smaller. Developers concerns like architecture and choice of libraries are not what drives an application. An application is driven by its functionality. If you have a useful application then licensing tends to be a secondary concern. The licensing tends to define the community in some sense as well, and as a result there’s a kind of opt-in consensus. And I think the GPL in these cases really does create an environment of collaboration around extensions. It has a real benefit.

This seems to be true empirically as well. GPL’d libraries don’t tend to get very popular. People will do crazy-big projects just because of licensing issues. GPL’d Applications seem to do quite well, with examples like WordPress, Plone, The Gimp, and even the Linux kernel, which is closer in structure to an application than a library.

I haven’t studied the GPL v3 very closely, but it seems useful to me. Software-as-a-service has the potential to make the GPL irrelevant (especially for the kind of code I write), and could put the GPL in the same category as the LGPL where it retains its stigma without offering any practical advantages. Version 3 makes the permission/GPL choice seem useful, but so far I’ve never made any truly conscious choice between version 2 and 3.

If you are thinking about choosing a license, or thinking about choosing software based on the licensing, maybe these thoughts will be helpful in your own thinking. And please don’t GPL-hate in the comments, I’m not interested and if you feel the need then go write your own post.

Update: I wrote a second article to expand on some thoughts: The GPL and Principles

Listening to Tavis Smiley’s show tonight, and the segment My America where they talked about gun violence. At one point they quote a man who lost his brother to gun violence:

You can be in a club and bump into somebody on accident, a little of your liquor, a little of your water spill on their coat, now, you go outside, he got five or six people out their because you spilled your damn drink. Which, a person should be able to say, "man, my fault dog, I apologize, you know how it is." You got people that just ain’t gonna be right, man.

Tavis Smiley: So you take that, you put guns into the equation, that changes mediation efforts dramatically.

Several times they talk about how small matters of respect lead to violence. The conclusion is that guns are the problem.

I don’t really know what to do with this. In my life (and I suspect all of your lives) issues of respect do not lead to violence. As a result I have a hard time thinking of this as a gun problem.

OK, so it’s a violence problem. The other thing that gets me is there’s this strong undertone to this conversation that "we aren’t doing enough." This attitude is of course the norm for an NPR show. But it’s not we — I, and everyone I know is not part of this we. My "we" does not resort to violence. My "we" does not project respect into minor social interactions. When I say it’s not "we", I don’t think it’s just that I tuned into the wrong radio show — am I being recruited into this "we"? Do they really think listeners are part of this "we"?

There is no reflection in these shows about why this (whatever the issue of the show) is a general problem. Of course most talk shows tend to generalize wildly, to turn every anecdote into a sign of some change in culture, some disease of our society, something more than just an anecdote. (Though some good NPR shows do not attempt to generalize anecdotes at all.)

There’s a strong attitude, in this show and others, that this is a problem for us all to solve. Why exactly is this a problem for me to solve? Why is this a problem for government to solve? (I’m not a conservative, but I feel it’s unfair that only conservatives seem to be able to ask that question: why should government solve this?)

I don’t ask these questions rhetorically (and maybe that makes me different from the conservatives, who tend to only ask questions rhetorically). There may be a good answer to these questions. But it’s far too easy to say "we must do something about this" without saying who and why. We (especially those of us who listen to NPR) are all far too fatigued with the constant admonitions that not enough is being done, and something has to change. This kind of approach is not an effective call to action.

And it’s yet another thing trying to make me feel bad for something that’s not my fault. And dammit, it really isn’t my fault!

My old do-it-yourself framework tutorial was getting a bit long in the tooth, so I rewrote it to use WebOb. Now: the new do-it-yourself framework.

So I promised some more technical discussion of App Engine than my last two posts. Here it is:

Google App Engine uses a somewhat CGI-like model. That is, a script is run, and it uses stdin/stdout/environ to handle the requests. To avoid the overhead of CGI a process can be reused by defining __main__.main(). But while a process can be reused, it might not be, and of course it might get run on an entirely separate server. So in many ways it’s like the CGI model, with a small optimization so that, particularly under load, your requests can run with less latency.

This part is all well and good. I’ve already come to terms with servers going up and down without warning. But the environment itself has a number of other restrictions. It seems that App Engine is providing security in the language itself. The interpreter has been modified so that code is sandboxed, with no ability to write to the disk, open sockets, import C extensions, and see quite a few things in its environment. It’s these things that are a bit harder to come to terms with.

While they claim it supports any Python framework, these restrictions don’t actually make it easy. So for the last few days quite a few of us have been hacking various things to get stuff working.

The first thing people noticed is that Mako and Genshi didn’t work, because they use the ast (via the parser module) to handle the templating, and that module has been restricted. Apparently arbitrary bytecode is not safe in this environment, and so anything that can produce bytecode is considered dangerous. From what I understand Philip Jenvy has been working on Mako and the trunk is currently working. He’d already been doing work to get Mako working on Jython, which had similar issues. Genshi is also in progress and fairly close to working, though with some missing features. Genshi has the harder task as Mako was primarily reading the ast, while Genshi was writing it.

The first thing I noticed is that Setuptools doesn’t work. I’m flattered that one of the only 3 libraries included with App Engine is WebOb, but of course I am more enamored of a rich set of reusable libraries. Setuptools didn’t work because several modules and functions have been removed — this like os.open, os.uname, imp.acquire_lock, etc. Some of these are kind of reasonable, while others are not. The removal of many functions from imp doesn’t really make sense, for instance (I think the motivation was the difficulty of auditing the implementation of those functions, not that the functionality itself is dangerous). And while some functions can’t be used in the environment, the fact you can’t import those functions is more problematic. For instance, The Setuptools’ pkg_resources module has support to unzip eggs when they are imported. App Engine doesn’t support importing from zip files at all, and you certainly can’t unzip to a temporary location. But withoutthe necessary modules and objects pkg_resources won’t even import.

To work around this I started a new project: appengine-monkey, which adds several monkeypatches and replacement dummy modules to the environment to simulate a more typical environment. It’s just a small list so far (mostly in this module), but I expect as people experiment with other libraries the list will increase. For example, I would welcome implementations of things like httplib on top of urlfetch in this library. (Implementing httplib and stubbing out parts of socket would probably make urllib run.)

But the good news is that Pylons is pretty much working on App Engine, as is Setuptools and you can manage your libraries using virtualenv.

The instructions are all located in the appengine-monkey Pylons wiki page. Please leave comments if you have improvements or problems with that process. I also welcome contributors and developers to the project itself — this is a project for expediting App Engine development, it is not a project I care to champion or control. Or support to any large degree.

One ticket which is rather important is the apparent maximum number of files and blobs: 1000. Libraries involve lots of files, and the base Pylons install is only barely under this limit. Now I just wish I could use lxml, but that’s probably going to be a long time coming.