1. Debian Python packaging toolchains reuse Distutils. As a Python developper, I want to write a setup.py; Debian wants a makefile. Well, the debian/rules makefile uses the setup.py script with appropriate command-line options. (Dependencies in the Python system would be a must though, for now they are in a Debian-specific file.) See, no conflict between the two systems.

2. Integrating PyPI and APT isn’t just a technical issue: Debian packages are DFSG-free. (Of course, Debian means “main”-only.) I am totally in favor of decentralisation for VCS for instance, but I like the centralized control of the freeness of software allowed by using Debian archives.

3. APT can work with local repositories (i.e. lines starting with “deb file:///” in the sources.list file). A local repository (or an HTTP one, for that matter) is really a bunch of deb packages and a few meta-files. Automation can be done with the help of mini-dinstall, pbuilder, dnotify, *-buildpackage. (After all, pip is “just” automation of such things ;)

Well, I know my points only work if you’re already happily using a deb-based system. I don’t know whether and how Python developpers should compensate Windows flaws. And if you like setup.py develop, I don’t know how much coding would be required to automate these steps using a local APT repository, but that would be worth trying.

Cheers, Merwok

Python is a general purpose language. But here everybody just seems to be talking about web development!

In web development you’ve got to distinguish the users from the consumers. The consumers do not do any administration, that’s only the users who are often/mostly also the administrators of that server/host/machine. There one can much more easily get away with pulling up an alternative management system in parallel to the distribution’s own system without doing any major harm.

But with most other applications of Python it’s a bit different. Whenever you use Python for system administrative tools (Ubuntu has got quite a few already), for libraries/bindings to libraries, or for application level (excluding web applications) it’s all a different ball game. In all these cases the system’s package management becomes much more important than Python’s own attempts to cater for the developer and/or web admin needs.

Boiling down to this: I can fully understand both sides. Unfortunately there’s quite some clash between them as described. The challenges ahead would be now to find a Good Solution (TM) that is well suitable to bridge the gap and cater for the non-web needs. Maybe even some sort of a PyPI mirror that offers eggs as pre-rolled .deb/.rpm packages for different distributions.

jorge: Distributions handle the CherryPy2/3 issue with various means. last I checked in Debian they have an explicit Conflict so that CherryPy3 and Turbogears-1 apps cannot be installed together on the same box.

so either kill off all the TG1 community of the CP3 community, that isn't really nice. How do you handle zope2/zope3? now let's go to the small details. 
MY app needs version 2.3.1 of library X, your app needs 2.3.2 yet for some unknown issue they are incompatible. WHy do I have to choose which system to use?

In Fedora we’ve made use of the multiple version functionality of setuptools (and run up against limitations in its design) so that you can have both versions of CherryPy installed at the same time.

That's great but keep in mind a package needs to be fairly popular for this to work (GTK,python,etc) what about something less known by the public 
like pylons0.9.6/0.9.7 (and yes apps in .6 wont run out of the box on .7).

Note that for a distribution POV, we’d love to have a versioning scheme that works flawlessly. Debian, where the idea of doing compatible python modules was rejected, versions each of their C library packages so you can have separate versions. Fedora, which doesn’t like to do them for maintainability reasons (what do we do when a security issue arises in CherryPy2 and the devs say, “CP2 is no longer supported, we’re only releasing a security fix for CherryPy3″?), will do compat packages like the CherryPy2 package when the value for keeping the apps outweighs the maintainance costs. ATM, only strategies like psycopg use (changing the module name from psycopg to psycopg2 on a major, API changing update) work 100%. The setuptools method has promise, though… if only it would mature a bit more.

Well that is a risk some people have to take, and is why not all applications are packaged, because I totally agree that the maintainer of a package 
should be it's developer. Not several people over several OS's monkeypatching it to work on their OS, because it's important yet the original author is 
gone. Now this particular problem is not related to packager it's related to developers, if TG1 refuses to die. Even after CP people say no more fixes for 
CP2, then it's their responsibility to maintain CP2.

Let’s say you install trac using portage with emerge =trac-0.11.2, the files will just be placed into /usr/share/webapps/trac/0.11.2/. You will then use webapp-config to deploy it to the desired web server(s) or vhost(s).

It is far from perfect (e.g. doesn’t support virtualenv, doesn’t support nginx), but at least it partially solves the problem where more than one copy and/or more than one version of a webapp are required to be running on the same machine.

Btw, it is also “backward-compatible” with other less capable packaging tools, such that you can configure it to auto-deploy to your localhost whenever you emerge a webapp package, if you only need just one copy and one version of that webapp.

Anyway, to get anything done at work, we just use a combination of virtualenv, custom distcmds, and a local eggbasket. A great language and platform independent package manager that can work throughout the development and deployment lifecycle would certainly be very welcome :)

1) Distributions are an attempt to freeze the world in a one entirely self consistent image. Packager’s priorities are completeness (no broken dependencies), self consistency, and stability. That’s why they prefer to package the latest stable version. But many times, the stable version of one particular program depends on the unstable version of a library (that’s only the first problem).

2) Packagers are also concerned about space, due to disk and bandwidth concerns. That means eliminating redundancy. Most packagers hate the idea of having multiple versions of a single library “just because” one app needs it.

3) Last but not least, packagers are concerned about security. Again, having multiple versions of a library tends to make their work harder due to the potential combinations and security threats.

That’s not a problem for conventional applications, because most applications evolve slowly, and even when that’s not the case, most users do not need the latest upgrades - unless there’s a security issue, which are addressed by minimal patches that only fix the problem and do not add any functionality (so dependencies keep relatively unchanged).

Development is about bringing new functionality at the fastest pace possible. It means that:

• Developers are comfortable about using the latest libraries, even if they are not considered stable (and that leads to problem #1 mentioned above).

• Developers need a completely different environment to work than the one recommended to run the final application.

To make things more confusing, the world is not black and white. There’s a middle ground between the two. Trac, for instance - most of the times I can work with a older version, so I don’t mind installing the packaged one. And it will work out of the box for most users. But for other packages, it’s much harder to come up with a sensible installation out of a generic package.

One of my particular gripes is with documentation. Take some package (for example, Twiki or Plone). Install it and try to customize it, following the instructions on the original website. Most of the times*, there are no instructions on how to install the “packaged version”, and some of the instructions that apply to the “tgz version” do not make sense or are not applicable to the packaged one. Locations differ, some of the parameters are already customized, and that just makes life more difficult for the users.

(*) In some cases the instructions are there but are incomplete or out of date.

This as an admission that the packaging tools are NOT adequate for developer needs. Why does having isolated environments preclude simple administration? If the package management tools were in charge of managing environments, then they would know where to check for installed package versions, even if I had hundreds scattered about my home directory.

There’s no reason Trac or any other application needs to depend on separately packaged Python libraries, and I do think that’s specifically problematic. But if yum install mysite doesn’t involve any dependencies (except for very conservative packages) then that could work pretty well. I’d still be pretty worried about migrations, though… but arguably developers should work harder at making data migration work with less interaction. Having specific support for data migration (mostly meaning clear policies for how backup and migration and reverts should work) would also be an improvement for people creating packages.

Of course it’s also common for web developers to have two versions of the exact same application simultaneously deployed on a single box. I find the way Linux package managers deal with versions to simply be… dumb. As in not-technically-advanced-in-a-way-that-actively-hurts. There’s also problems about the development build process being very different from the deployment process, which leads to bugs in one or the other. These bugs cause a lot of social problems, as they aggravate the already difficult relationship between sysadmin and developer (a relationship that can be difficult even when they are the same person ;).

What we need is a great language and platform independent package manager that can work throughout the development and deployment lifecycle. I know of no such thing at the moment. (I thought rpath might be like that, but from what I can tell it is not.)

In Fedora we’ve made use of the multiple version functionality of setuptools (and run up against limitations in its design) so that you can have both versions of CherryPy installed at the same time.

Note that for a distribution POV, we’d love to have a versioning scheme that works flawlessly. Debian, where the idea of doing compatible python modules was rejected, versions each of their C library packages so you can have separate versions. Fedora, which doesn’t like to do them for maintainability reasons (what do we do when a security issue arises in CherryPy2 and the devs say, “CP2 is no longer supported, we’re only releasing a security fix for CherryPy3″?), will do compat packages like the CherryPy2 package when the value for keeping the apps outweighs the maintainance costs. ATM, only strategies like psycopg use (changing the module name from psycopg to psycopg2 on a major, API changing update) work 100%. The setuptools method has promise, though… if only it would mature a bit more.

There’s also a lot of conflicts between Setuptools and package
maintainers. This is kind of a proxy war between developers and sysadmins,
who have very different motivations.[1]_

If everyone would keep in mind that there’s more than one audience for different stages in an app’s lifetime, then I think even Rudd-O and you could at least see how your views interlock.

.. _[1]: Note that instead of seeing this as a war by proxy, I look at it as a contract negotiation with a mediator. The two parties involved are system administrators and developers. The poor packagers are the mediator that has to carry requirements and messages from one side to the other.

The Developer

$ virtualenv mysite
$ cd mysite
$ svn co http//mycompany.com/svn/configs/mysite etc/
$ pip -E . install -r etc/requirements.txt
$ ./bin/paster serve etc/development.ini

There’s many many ways to make this work using packaging tools. However, the appropriate methods for the different audiences are not the same. If the audience is the developer who has just installed a new workstation and needs to start working on the next generation of the website using virtualenv is the correct thing to do. However, what it’s doing is creating a customized environment for running a particular environment. This is anathema to making a system administrators job easy.

The Deployment Engineer

$ yum install --disable-repo='*' --enable-repo='rhel-6*' --installroot=/srv/testbed/ mysite-0.1-0.noarch.rpm
$ cd /srv/testbed/
$ rm -rf mysite
$ svn co http//mycompany.com/svn/applications/mysite
$ PYTHONPATH=/srv/testbed/usr/lib/python2.5:/srv/testbed/usr/lib/python2.5/site-packages:"$PYTHONPATH"

After the developer has gotten the application to a certain level of functionality some engineer has to start making sure it runs the platforms the company cares to support (either internally or externally). They can create a local install in a directory using the platform’s package repositories and checkin changes to the application code to support that platform. In open source, the Developer and Packager often share the Deployment Engineer role.

The Packager

$ wget http//mycompany.com/releases/mysite-1.0.tar.gz
$ vim mysite.spec
$ rpmbuild mysite.spec

The packager may be part of mycompany or a Linux distribution but they are building the software specifically for a distribution and therefore care about the conventions that a distribution uses to make their users’ (system administrators) lives easier.

The System Administrator

# yum install mysite

The ideal for the system administrator is to run a single command that downloads and installs the application and all its dependencies. By having the application inside a distro package they also have the ability to do these things:

Check what hotfixes were applied:
# rpm -V mysite mydependency
Check the version:
# rpm -q mysite mydependency
Get security updates for mysite, dependencies, and the whole server:
# yum update
Install the same setup onto a new server
# yum install mysite

Notes

Not all applications have all audiences explicitly. There’s many web applications that are not meant to be distributed, for instance. However, even in a one-company-only application, it’s important to remember that the system administrator audience exists even if it is the same person as the developer. If your web site becomes popular, you may need to start load balancing it. Your current server may go belly up. The operating system you’re using may reach its end of life and you have to port to a new version. You may leave the company or your company may grow and hire a system admin.

[ed note: URLs mangled to keep WordPress from mangling them more]