Software packaging and deployment for complex sites

There are many different software packaging formats, but few that cater for the needs of large or complex software deployment environments.

Some time ago I inherited a problem or rather three hundred of them to be precise. Servers. Nice ones, shiny ones, unix ones. There were no problems…as long as we didn’t install software on them. Not that there was anything wrong with our software per-se but the deployment systems were a little primitive (think tar and friends). Combine that with the pressure of deadlines and the insatiable urge to add “just one more feature” after testing and… well…there were problems.

With my wishlist in one hand and a large cup of coffee in the other I set off in search for the packaging system to use, but after hunting I came to the conclusion there wasn’t a system that met my needs. My main needs included:

  • centrally managed “push” software deployment
  • per target/class software deployment (ie different software on different targets)
  • guaranteed software installation
  • installed package verification and repair
  • version and dependency management
  • optimal use of network bandwidth

Although bandwidth is less of an issue these days, it is still not something to be squandered. One of the limitations of most packaging systems is that they use a single (often compressed) file for the package. Whilst this makes sense at many levels it does make it hard to be bandwidth efficient – even a single byte change in a file normally requires the entire package (often many megabytes) to be downloaded.

(I believe in elegant and efficient design first and throwing more CPU/RAM/Disk/Bandwidth at it as a secondary solution. Elegance as a concept cuts across many domains including simplicity of concept and expression. It doesn’t always mean minimal bandwidth for example, but if you’ve come from a real-time or limited resources background you’ll understand the origins of my viewpoint.)

The problem divides naturally into four interacting parts:

There are three toolsets I built to address these problems

  • kpkg (packaging)
  • pacman (distribution management)
  • pup (distribution, installation and management)