Software Packaging and Deployment

Software packaging and deployment for complex sites

There are many different software packaging formats, but few that cater for the needs of large or complex software deployment environments.

Some time ago I inherited a problem or rather three hundred of them to be precise. Servers. Nice ones, shiny ones, unix ones. There were no problems…as long as we didn’t install software on them (or administer them). Not that there was anything wrong with our software per-se but the deployment systems were a little primitive (think tar and friends). Combine that with the pressure of deadlines and the insatiable urge to add “just one more feature” after testing and… well…there were problems.

With my wishlist in one hand and a large cup of coffee in the other I set off in search for the packaging system to use, but after hunting I came to the conclusion there wasn’t a system that met my needs which included:

  • centrally managed “push” software deployment
  • per target/class software deployment (ie different software on different targets)
  • guaranteed software installation
  • installed package verification and repair
  • version and dependency management
  • optimal use of network bandwidth

Although bandwidth is less of an issue these days, it is still not something to be squandered when it costs money. One of the limitations of most packaging systems is that they use a single (often compressed) file for the package. Whilst this makes sense at many levels it does make it hard to be bandwidth efficient – even a single byte change in a file normally requires the entire package (often many megabytes) to be downloaded.

I believe in elegant and efficient design first and throwing more CPU/RAM/Disk/Bandwidth at it as a secondary solution. Elegance as a concept cuts across many domains including simplicity of concept and expression. It doesn’t always mean minimal bandwidth for example, but if you’ve come from a real-time or limited resources background you’ll understand the origins of my viewpoint.

The problem divides naturally into four interacting parts:

There are three toolsets I built to address these problems

kpkg (packaging)
pacman (distribution management)
pup (distribution, installation and management)

Whilst pacman and pup use kpkg format, pacman in particular is designed to be package format agnostic, and so should be useable with debian, red hat and YFPS (your favourite packaging system).
Pacman in particular is well tested and has been used for 5 years in a complex commercial environment.

  • + : A leading plus sign indicates that this word must be present in every object returned.
  • - : A leading minus sign indicates that this word must not be present in any row returned.
  • By default (when neither plus nor minus is specified) the word is optional, but the object that contain it will be rated higher.
  • < > : These two operators are used to change a word's contribution to the relevance value that is assigned to a row.
  • ( ) : Parentheses are used to group words into subexpressions.
  • ~ : A leading tilde acts as a negation operator, causing the word's contribution to the object relevance to be negative. It's useful for marking noise words. An object that contains such a word will be rated lower than others, but will not be excluded altogether, as it would be with the - operator.
  • * : An asterisk is the truncation operator. Unlike the other operators, it should be appended to the word, not prepended.
  • " : The phrase, that is enclosed in double quotes ", matches only objects that contain this phrase literally, as it was typed.


Related Sites