Personal tools
You are here: Home → Documentation → Tutorials → Introduction to zc.buildout
 

Introduction to zc.buildout

Note: Return to tutorial view.

Jim Fulton's tutorial for using buildout, originally given at DZUG 2007

What is zc.buildout

zc.buildout is a coarse-grained python-based configuration-driven build tool

What is zc.buildout?

  • Coarse-grained python-based configuration-driven build tool

  • Tool for working with eggs

  • Repeatable

    handout

    It should be possible to check-in a buildout specification and reproduce the same software later by checking out the specification and rebuilding.

  • Developer oriented

Coarse-grained building

  • make and scons (and distutils) are fine grained

    • Focus on individual files

    • Good when one file is computed from another

      .c -> .o -> .so

    • rule-driven

    • dependency and change driven

  • zc.buildout is coarse-grained

    • Build large components of a system
      • applications
      • configurations files
      • databases
    • configuration driven

Python-based

  • make is an awful scripting language

    • uses shell
    • non-portable

  • Python is a good scripting language

    handout

    Fortunately, distutils addresses most of my building needs. If I had to write my own fine-grained build definition, I'd use scons.

Working with eggs

Python eggs and how zc.buildout gives you better control over using eggs
  • Eggs rock!
  • easy_install
    • Easy!
    • Installs into system Python
    • Not much control
  • workingenv makes easy_install much more usable
    • Avoids installing into system Python
    • Avoids conflicts with packages installed in site_packages
    • Really nice for experimentation
    • Easy!
    • Not much control

zc.buildout and eggs

  • Control

    • Configuration driven

      • easier to control versions used

      • always look for most recent versions by default

        handout

        When upgrading a distribution, easy_install doesn't upgrade dependencies,

      • support for custom build options

  • Greater emphasis on develop eggs

    • Automates install/uninstall
    • preference to develop eggs
    handout

    I often switch between develop and non-develop eggs. I may be using a regular egg and realize I need to fix it. I checkout the egg's project into my buildout and tell buildout to treat it as a develop egg. It creates the egg link in develop eggs and will load the develop egg in preference to the non-develop egg.

    (easy_install gives preference to released eggs of the same version.)

    When I'm done making my change, I make a new egg release and tell buildout to stop using a develop egg.

handout
zc.buildout is built on setuptools and easy_install.

zc.buildout current status

  • Actively used for development and deployment

  • Third-generation of ZC buildout tools

    handout

    Our earliest buildouts used make. These were difficult to maintain and reuse.

    Two years ago, we created a prototype Python-based buildout system.

    zc.buildout is a non-prototype system that reflects experience using the prototype.

  • A number of "recipes" available

A Python Egg Primer

Eggs are simple!

  • directories to be added to path

    • may be zipped
    • "zero" installation

  • Meta data

    • dependencies
    • entry points

  • May be distributed as source distributions

    handout

    easy_install and zc.buildout can install source distributions as easily as installing eggs. I've found that source distributions are more convenient to distribute in a lot of ways.

  • Automatic discovery through PyPI

Glossary of Egg Jargon

Terminology used when refering to eggs

Egg jargon

  • Distribution

    handout

    "distribution" is the name distutils uses for something that can be distributed. There are several kinds of distributions that can be created by distutils, including source distributions, binary distributions, eggs, etc.

  • source and binary distributions

    handout

    A source distribution contains the source for a project.

    A binary distributions contains a compiled version of a project, including .pyc files and built extension modules.

    Eggs are a type of binary distribution.

  • Platform independent and platform dependent eggs

    handout

    Platform dependent eggs contain built extension modules and are thus tied to a specific operating system. In addition, they may depend on build options that aren't reflected in the egg name.

  • develop egg links

    handout

    Develop egg links (aka develop eggs) are special files that allow a source directory to be treated as an egg. An egg links is a file containing the path of a source directory.

  • requirements

    handout

    Requirements are strings that name distributions. They consist of a project name, optional version specifiers, and optional extras specifiers. Extras are names of features of a package that may have special dependencies.

  • index and link servers

    easy_install and zc.buildout will automatically download distributions from the Internet. When looking for distributions, they will look on zero or more links servers for links to distributions.

    They will also look on a single index server, typically (always) http://www.python.org/pypi. Index servers are required to provide a specific web interface.

Entry points

  • Very similar to utilities

    • Named entry point groups define entry point types
    • Named entry points within groups provide named components of a given type.

  • Allow automated script generation

    Wrapper script:

    • Sets up path

      handout

      easy_install and zc.buildout take very different approaches to this.

      easy_install generates scripts that call an API that loads eggs dynamically at run time.

      zc.buildout determines the needed eggs at build time and generates code in scripts to explicitly add the eggs to sys.path.

      The approach taken by zc.buildout is intended to make script execution deterministic and less susceptible to accidental upgrades.

    • Imports entry point

    • Calls entry point without arguments

      handout

      Buildout allows more control over script generation. Initialization code and entry point arguments can be specified.

Buildout overview

Define the full system using ConfigParser format extended with a variable-substitution syntax

  • Configuration driven

    • ConfigParser +

      handout

      Buildout uses the raw ConfigParser format extended with a variable-substitution syntax that allows reference to variables by section and option:

      ${sectionname:optionname}

    • Allows full system to be defined with a single file

      handout

      Although it is possible and common to factor into multiple files.

  • Specify a set of "parts"

    • recipe
    • configuration data
    handout

    Each part is defined by a recipe, which is Python software for installing or uninstalling the part, and data used by the recipe.

  • Install and uninstall

    handout

    If a part is removed from a specification, it is uninstalled.

    If a part's recipe or configuration changes, it is uninstalled and reinstalled.

Buildout overview (continued)

  • Recipes
    • Written in python
    • Distributed as eggs
  • Egg support
    • Develop eggs
    • Egg-support recipes

Quick intro

  • Most common case
    • Working on a package
    • Want to run tests
    • Want to generate distributions
  • buildout is source project
  • Example: zope.event

zope.event example

Example buildout file for working on the zope.event egg

zope.event project files

  • source in src directory

    handout

    Placing source in a separate src directory is a common convention. It violates "shallow is better than nested". Smaller projects may benefit from putting sources in the root directory,

  • setup.py for defining egg

    handout

    Assuming that the project will eventually produce an egg, we have a setup file for the project. As we'll see later, this can be very minimal to start.

  • README.txt

    handout

    It is conventional to put a README.txt in the root of the project. distutils used to complain if this wasn't available.

  • bootstrap.py for bootstrapping buildout

    handout

    The bootstrap script makes it easy to install the buildout software. We'll see another way to do this later.

  • buildout.cfg defines the buildout

zope.event buildout.cfg

[buildout]
parts = test
develop = .

[test]
recipe = zc.recipe.testrunner
eggs = zope.event
handout

Let's go through this line by line.

[buildout]

defines the buildout section. It is the only required section in the configuration file. It is options in this section that may cause other sections to be used.

parts = test

Every buildout is required to specify a list of parts, although the parts list is allowed to be empty. The parts list specifies what to build. If any of the parts listed depend on other parts, then the other parts will be built too.

develop = .

The develop option is used to specify one or more directories from which to create develop eggs. Here we specify the current directory. Each of these directories must have a setup file.

[test]

The test section is used to define our test part.

recipe = zc.recipe.testrunner

Every part definition is required to specify a recipe. The recipe contains the Python code with the logic to install the part. A recipe specification is a distribution requirement. The requirement may be followed by an colon and a recipe name. Recipe eggs can contain multiple recipes and can also define an default recipe.

The zc.recipe.testrunner egg defines a default recipe that creates a test runner using the zope.testing.testrunner framework.

eggs = zope.event

The zc.recipe.testrunnner recipe has an eggs option for specifying which eggs should be tested. The generated test script will load these eggs along with their dependencies.

For more information on the zc.recipe.testrunner recipe, see http://www.python.org/pypi/zc.recipe.testrunner.

Buildout steps

Typically actions taken when working with buildout

Buildout steps

  • Bootstrap the buildout:

    python bootstrap.py
    handout

    This installs setuptools and zc.buildout locally in your buildout. This avoids changing your system Python.

  • Run the buildout:

    bin/buildout
    handout

    This generates the test script, bin/test.

  • Run the tests:

    bin/test

  • Generate a distribution:

    bin/buildout setup . sdist register upload
bin/buildout setup . bdist_egg register upload

    bin/buildout setup . egg_info -rbdev sdist register upload
    handout

    Buildout accepts a number of commands, one of which is setup. The setup command takes a directory name and runs the setup script found there. It arranges for setuptools to be imported before the script runs. This causes setuptools defined commands to work even for distributions that don't use setuptools.

    The sdist, register, upload, bdist_egg, and egg_info commands are setuptools and distutils defined commands.

    The sdist command causes a source distribution to be created.

    The register command causes a release to be registered with PyPI and the upload command uploads the generated distribution. You'll need to have an account on PyPI for this to work, but these commands will actually help you set an account up.

    The bdist_egg command generates an egg.

    The egg_info command allows control of egg meta-data. The -r option to the egg_info command causes the distribution to have a version number that includes the subversion revision number of the project. The -b option specified a revision tag. Here we specified a revision tag of "dev", which marks the release as a development release. These are useful when making development releases.

Excersize 1: zope.event

Perform your first buildout exercise

Exercise 1

handout
We won't have time to stop the lecture while you do the exercises. If you can play and listen at the same time, then feel free to work on them while I speak. Otherwise, I recommend doing them later in the week. Feel free to ask me questions if you run into problems.

Try building out zope.event.

  • Check out: svn://svn.zope.org/repos/main/zope.event/trunk
  • Bootstrap
  • Run the buildout
  • Run the tests
  • Look around the buildout to see how things are laid out.
  • Look at the scripts in the bin directory.

buildout layout and common use cases

Buildout directory structure and simple, common use casese

buildout layout

  • bin directory for generated scripts

  • parts directory for generated part data

    Many parts don't use this.

  • eggs directory for (most) installed eggs

    • May be shared across buildouts.

  • develop-eggs directory

    • develop egg links
    • custom eggs

  • .installed.cfg records what has been installed

handout

Some people find the buildout layout surprising, as it isn't similar to a Unix directory layout. The buildout layout was guided by "shallow is better than nested".

If you prefer a different layout, you can specify a different layout using buildout options. You can set these options globally so that all of your buildouts have the same layout.

Common buildout use cases

  • Working on a single package

    handout

    zope.event is an example of this use case.

  • System assembly

  • Try out new packages

    • workingenv usually better
    • buildout better when custom build options needed

  • Installing egg-based scripts for personal use

    ~/bin directory is a buildout

Creating eggs

Three levels of egg development

  • Develop eggs, a minimal starting point
  • Adding data needed for distribution
  • Polished distributions

A Minimal/Develop setup.py

from setuptools import setup
setup(
    name='foo',
    package_dir = {'':'src'},
    )
handout

If we're only going to use a package as a develop egg, we just need to specify the project name, and, if there is a separate source directory, then we need to specify that location.

We'd also need to specify entry points if we had any. We'll see an example of that later.

See the setuptools and distutils documentation for more information.

Distributable setup.py

from setuptools import setup, find_packages
name='zope.event'
setup(
    name=name,
    version='3.3.0',
    url='http://www.python.org/pypi/'+name,
    author='Zope Corporation and Contributors',
    author_email='zope3-dev@zope.org',
    package_dir = {'': 'src'},
    packages=find_packages('src'),
    namespace_packages=['zope',],
    include_package_data = True,
    install_requires=['setuptools'],
    zip_safe = False,
    )
handout

If we want to be able to create a distribution, then we need to specify a lot more information.

The options used are documented in either the distutils or setuptools documentation. Most of the options are fairly obvious.

We have to specify the Python packages used. The find_packages function can figure this out for us, although it would often be easy to specify it ourselves. For example, we could have specified:

packages=['zope', 'zope.event'],

The zope package is a namespace package. This means that it exists solely as a container for other packages. It doesn't have any files or modules of it's own. It only contains an __init__ module with:

pkg_resources.declare_namespace(__name__)

or, perhaps:

# this is a namespace package
try:
    import pkg_resources
    pkg_resources.declare_namespace(__name__)
except ImportError:
    import pkgutil
    __path__ = pkgutil.extend_path(__path__, __name__)

Namespace packages have to be declared, as we've done here.

We always want to include package data.

Because the __init__ module uses setuptools, we declare it as a dependency, using install_requires.

We always want to specify whether a package is zip safe. A zip safe package doesn't try to access the package as a directory. If in doubt, specify False. If you don't specify anything, setuptools will guess.

Polished setup.py (1/3)

import os
from setuptools import setup, find_packages

def read(*rnames):
    return open(os.path.join(os.path.dirname(__file__), *rnames)).read()

long_description=(
        read('README.txt')
        + '\n' +
        'Detailed Documentation\n'
        '**********************\n'
        + '\n' +
        read('src', 'zope', 'event', 'README.txt')
        + '\n' +
        'Download\n'
        '**********************\n'
        )

open('documentation.txt', 'w').write(long_description)
handout

In the polished version we flesh out the meta data a bit more.

When I create distributions that I consider ready for broader use and upload to PyPI, I like to include the full documentation in the long description so PyPI serves it for me.

Polished setup.py (2/3)

name='zope.event'
setup(
    name=name,
    version='3.3.0',
    url='http://www.python.org/pypi/'+name,
    license='ZPL 2.1',
    description='Zope Event Publication',
    author='Zope Corporation and Contributors',
    author_email='zope3-dev@zope.org',
    long_description=long_description,

    packages=find_packages('src'),
    package_dir = {'': 'src'},
    namespace_packages=['zope',],
    include_package_data = True,
    install_requires=['setuptools'],
    zip_safe = False,
    )

Extras

name = 'zope.component'
setup(name=name,
      ...
      namespace_packages=['zope',],
      install_requires=['zope.deprecation', 'zope.interface',
                        'zope.deferredimport', 'zope.event',
                        'setuptools', ],
      extras_require = dict(
          service = ['zope.exceptions'],
          zcml = ['zope.configuration', 'zope.security', 'zope.proxy',
                  'zope.i18nmessageid',
                  ],
          test = ['zope.testing', 'ZODB3',
                  'zope.configuration', 'zope.security', 'zope.proxy',
                  'zope.i18nmessageid',
                  'zope.location', # should be dependency of zope.security
                  ],
          hook = ['zope.hookable'],
          persistentregistry = ['ZODB3'],
          ),
      )
handout

Extras provide a way to help manage dependencies.

A common use of extras is to separate test dependencies from normal dependencies. A package may provide other optional features that cause other dependencies. For example, the zcml module in zope.component adds lots of dependencies that we don't want to impose on people that don't use it.

Recipes to install eggs, scripts and custom interpreters

Commonly used recipes

zc.recipe.egg

Set of recipes for:

  • installing eggs
  • generating scripts
  • custom egg compilation
  • custom interpreters

See: http://www.python.org/pypi/zc.recipe.egg.

Installing eggs

[buildout]
parts = some-eggs

[some-eggs]
recipe = zc.recipe.egg:eggs
eggs = docutils
       ZODB3 <=3.8
       zope.event
handout

The eggs option accepts one or more distribution requirements. Because requirements may contain spaces, each requirement must be on a separate line. We used the eggs option to specify the eggs we want.

Any dependencies of the named eggs will also be installed.

Installing scripts

[buildout]
parts = rst2

[rst2]
recipe = zc.recipe.egg:scripts
eggs = zc.rst2
handout

If any of the named eggs have console_script entry points, then scripts will be generated for the entry points.

If a distribution doesn't use setuptools, it may not declare it's entry points. In that case, you can specify entry points in the recipe data.

Script initialization

[buildout]
develop = codeblock
parts = rst2
find-links = http://sourceforge.net/project/showfiles.php?group_id=45693

[rst2]
recipe = zc.recipe.egg
eggs = zc.rst2
       codeblock
initialization =
    sys.argv[1:1] = (
      's5 '
      '--stylesheet ${buildout:directory}/zope/docutils.css '
      '--theme-url file://${buildout:directory}/zope'
      ).split()
scripts = rst2=s5
handout

In this example, we omitted the recipe entry point entry name because the scripts recipe is the default recipe for the zc.recipe.egg egg.

The initialization option lets us specify some Python code to be included.

We can control which scripts get installed and what their names are with the scripts option. In this example, we've used the scripts option to request a script named s5 from the rst2 entry point.

Custom interpreters

The script recipe allows an interpreter script to be created.

[buildout]
parts = mypy

[mypy]
recipe = zc.recipe.egg:script
eggs = zope.component
interpreter = py

This will cause a bin/py script to created.

handout

Custom interpreters can be used to get an interactive Python prompt with the specified eggs and and their dependencies on sys.path.

You can also use custom interpreters to run scripts, just like you would with the usual Python interpreter. Just call the interpreter with the script path and arguments, if any.

Excercise 2

Add a custom interpreter to the zope.event from excercise 1

Exercise 2

  • Add a part to the zope.event project to create a custom interpreter.
  • Run the interpreter and verify that you can import zope.event.

Building custom eggs

Custom egg builds such as a distribution with compiled files

Custom egg building

[buildout]
parts = spreadmodule

[spreadtoolkit]
recipe = zc.recipe.cmmi
url = http://yum.zope.com/buildout/spread-src-3.17.1.tar.gz

[spreadmodule]
recipe = zc.recipe.egg:custom
egg = SpreadModule ==1.4
find-links = http://www.python.org/other/spread/
include-dirs = ${spreadtoolkit:location}/include
library-dirs = ${spreadtoolkit:location}/lib
rpath = ${spreadtoolkit:location}/lib
handout

Sometimes a distribution has extension modules that need to be compiled with special options, such as the location of include files and libraries, The custom recipe supports this. The resulting eggs are placed in the develop-eggs directory because the eggs are buildout specific.

This example illustrates use of the zc.recipe.cmmi recipe with supports installation of software that uses configure, make, make install. Here, we used the recipe to install the spread toolkit, which is installed in the parts directory.

Part dependencies

  • Parts can read configuration from other parts
  • The parts read become dependencies of the reading parts
    • Dependencies are added to parts list, if necessary
    • Dependencies are installed first
handout
In the previous example, we used the spread toolkit location in the spreadmodule part definition. This reference was sufficient to make the spreadtoolkit part a dependency of the spreadmodule part and cause it to be installed first.

Custom develop eggs

[buildout]
parts = zodb

[zodb]
recipe = zc.recipe.egg:develop
setup = zodb
define = ZODB_64BIT_INTS
handout
We can also specify custom build options for develop eggs. Here we used a develop egg just to make sure our custom build of ZODB took precedence over normal ZODB eggs in our shared eggs directory.

Writing your own recipes

Use the zc.buildout recipe API

Writing recipes

  • The recipe API

    • install

      • __init__

        handout

        The initializer is responsible for computing a part's options. After the initializer call, the options directory must reflect the full configuration of the part. In particular, if a recipe reads any data from other sections, it must be reflected in the options. The options data after the initializer is called is used to determine if a configuration has changed when deciding if a part has to be reinstalled. When a part is reinstalled, it is uninstalled and then installed.

      • install

        handout

        The install method installs the part. It is used when a part is added to a buildout, or when a part is reinstalled.

        The install recipe must return a sequence of paths that that should be removed when the part is uninstalled. Most recipes just create files or directories and removing these is sufficient for uninstalling the part.

      • update

        handout

        The update method is used when a part is already installed and it's configuration hasn't changed from previous buildouts. It can return None or a sequence of paths. If paths are returned, they are added to the set of installed paths.

    • uninstall

      handout

      Most recipes simply create files or directories and the built-in buildout uninstall support is sufficient. If a recipe does more than simply create files, then an uninstall recipe will likely be needed.

Install Recipes

mkdirrecipe.py:

import logging, os, zc.buildout

class Mkdir:

    def __init__(self, buildout, name, options):
        self.name, self.options = name, options
        options['path'] = os.path.join(
                              buildout['buildout']['directory'],
                              options['path'],
                              )
        if not os.path.isdir(os.path.dirname(options['path'])):
            logging.getLogger(self.name).error(
                'Cannot create %s. %s is not a directory.',
                options['path'], os.path.dirname(options['path']))
            raise zc.buildout.UserError('Invalid Path')
handout
  • The path option in our recipe is interpreted relative to the buildout. We reflect this by saving the adjusted path in the options.
  • If there is a user error, we:
    • Log error details using the Python logger module.
    • Raise a zc.buildout.UserError exception.

mkdirrecipe.py continued

def install(self):
    path = self.options['path']
    logging.getLogger(self.name).info(
        'Creating directory %s', os.path.basename(path))
    os.mkdir(path)
    return path

def update(self):
    pass
handout

A well-written recipe will log what it's doing.

Often the update method is empty, as in this case.

Uninstall recipes

servicerecipe.py:

import os

class Service:

    def __init__(self, buildout, name, options):
        self.options = options

    def install(self):
        os.system("chkconfig --add %s" % self.options['script'])
        return ()

    def update(self):
        pass

def uninstall_service(name, options):
    os.system("chkconfig --del %s" % options['script'])
handout
Uninstall recipes are callables that are passed the part name and the original options.

Buildout entry points

setup.py:

from setuptools import setup

entry_points = """
[zc.buildout]
mkdir = mkdirrecipe:Mkdir
service = servicerecipe:Service
default = mkdirrecipe:Mkdir

[zc.buildout.uninstall]
service = servicerecipe:uninstall_service
"""

setup(name='recipes', entry_points=entry_points)

Exercise 3

Write recipe that creates a file from source given in a configuration option.
  • Write recipe that creates a file from source given in a configuration option.
  • Try this out in a buildout, either by creating a new buildout, or by extending the zope.event buildout.

Buildout command line, default settings and extending configurations

Using the command-line, operating modes, default settings and extending configurations

Command-line options

Buildout command-line:

  • command-line options and option setting
  • command and arguments
bin/buildout -U -c rpm.cfg install zrs
handout

Option settings are of the form:

section:option=value

Any option you can set in the configuration file, you can set on the command-line. Option settings specified on the command line override settings read from configuration files.

There are a few command-line options, like -c to specify a configuration file, or -U to disable reading user defaults.

See the buildout documentation, or use the -h option to get a list of available options.

Buildout modes

  • newest
    • default mode always tries to get newest versions
    • Turn off with -N or buildout newest option set to false.
  • offline
    • If enabled, then don't try to do network access
    • Disabled by default
    • If enabled, turn off with -o or buildout offline option set to false.
handout

By default, buildout always tries to find the newest distributions that match requirements. Looking for new distributions can be very time consuming. Many people will want to specify the -N option to disable this. We'll see later how we can change this default behavior.

If you aren't connected to a network, you'll want to use the offline mode, -o.

~/.buildout/default.cfg

Provides default buildout settings (unless -U option is used):

[buildout]
# Shared eggs directory:
eggs-directory = /home/jim/.eggs
# Newest mode off, reenable with -n
newst = false

[python24]
executabe = /usr/local/python/2.4/bin/python

[python25]
executabe = /usr/local/python/2.5/bin/python
handout

Unless the -U command-line option is used, user default settings are read before reading regular configuration files. The user defaults are read from the default.cfg file in the .buildout subdirectory of the directory specified in the HOME environment variable, if any.

In this example:

  • I set up a shared eggs directory.
  • I changed the default mode to non-newest so that buildout doesn't look for new distributions if the distributions it has meet it's requirements. To get the newest distributions, I'll have to use the -n option.
  • I've specified Python 2.4 and 2.5 sections that specify locations of Python interpreters. Sometimes, a buildout uses multiple versions of Python. Many recipes accept a python option that specifies the name of a section with an executable option specifying the location of a Python interpreter.

Extending configurations

The extends option allows one configuration file to extend another.

For example:

  • base.cfg has common definitions and settings

  • dev.cfg adds development-time options:

    [buildout]
extends = base.cfg

...

  • rpm.cfg has options for generating an RPM packages from a buildout.

Bootstrapping from existing buildout

  • The buildout script has a bootstrap command
  • Can use it to bootstrap any directory.
  • Much faster than running bootstrap.py because it can use an already installed setuptools egg.

Example: ~/bin directory

[buildout]
parts = rst2 buildout24 buildout25
bin-directory = .

[rst2]
recipe = zc.recipe.egg
eggs = zc.rst2

[buildout24]
recipe = zc.recipe.egg
eggs = zc.buildout
scripts = buildout=buildout24
python = python24

[buildout25]
recipe = zc.recipe.egg
eggs = zc.buildout
scripts = buildout=buildout25
python = python25
handout

Many people have a personal scripts directory.

I've converted mine to a buildout using a buildout configuration like the one above.

I've overridden the bin-directory location so that scripts are installed directly into the buildout directory.

I've specified that I want the zc.rst2 distribution installed. The rst2 distribution has a generalized version of the restructured text processing scripts in a form that can be installed by buildout (or easy_install).

I've specified that I want buildout scripts for Python 2.4 and 2.5. (In my buildout, I also create one for Python 2.3.) These buildout scripts allow me to quickly bootstrap buildouts or to run setup files for a given version of python. For example, to bootstrap a buildout with Python 2.4, I'll run:

buildout24 bootstrap

in the directory containing the buildout. This can also be used to convert a directory to a buildout, creating a buildout.cfg file is it doesn't exist.

zc.sharing example

A small example of the "system assembly" use case. We define a Zope 3 instance, and a test script.

Example: zc.sharing (1/2)

[buildout]
develop = . zc.security
parts = instance test
find-links = http://download.zope.org/distribution/

[instance]
recipe = zc.recipe.zope3instance
database = data
user = jim:123
eggs = zc.sharing
zcml =
  zc.resourcelibrary zc.resourcelibrary-meta
  zc.sharing-overrides:configure.zcml zc.sharing-meta
  zc.sharing:privs.zcml zc.sharing:zope.manager-admin.zcml
  zc.security zc.table zope.app.securitypolicy-meta zope.app.twisted
  zope.app.authentication
handout

This is a small example of the "system assembly" use case. In this case, we define a Zope 3 instance, and a test script.

You can largely ignore the details of the Zope 3 instance recipe. If you aren't a Zope user, you don't care. If you are a Zope user, you should be aware that much better recipes have been developped.

This project uses multiple source directories, the current directory and the zc.security directory, which is a subversion external to a project without its own distribution. We've listed both in the develop option.

We've requested the instance and test parts. We'll get other parts installed due to dependencies of the instance part. In particular, we'll get a Zope 3 checkout because the instance recipe refers to the zope3 part. We'll get a database part because of the reference in the database option of the instance recipe.

The buildout will look for distributions at http://download.zope.org/distribution/.

Example: zc.sharing (2/2)

[zope3]
recipe = zc.recipe.zope3checkout
url = svn://svn.zope.org/repos/main/Zope3/branches/3.3

[data]
recipe = zc.recipe.filestorage

[test]
recipe = zc.recipe.testrunner
defaults = ['--tests-pattern', 'f?tests$']
eggs = zc.sharing
       zc.security
extra-paths = ${zope3:location}/src
handout

Here we see the definition of the remaining parts.

The test part has some options we haven't seen before.

  • We've customized the way the testrunner finds tests by providing some testrunner default arguments.
  • We've used the extra-paths option to tell the test runner to include the Zope 3 checkout source directory in sys.path. This is not necessary as Zope 3 is now available entirely as eggs.

Source, Binary and RPM experiments

Distributing your buildout as source or binary, experiments with the RPM format

Source vs Binary

  • Binary distributions are Python version and often platform specific

  • Platform-dependent distribution can reflect build-time setting not reflected in egg specification.

    • Unicode size
    • Library names and locations

  • Source distributions are more flexible

  • Binary eggs can go rotten when system libraries are upgraded

    handout

    Recently, I had to manually remove eggs from my shared eggs directory. I had installed an operating system upgrade that caused the names of open-ssl library files to change. Eggs build against the old libraries no-longer functioned.

RPM experiments

Initial work creating RPMs for deployment in our hosting environment:

  • Separation of software and configuration
  • Buildout used to create rpm containing software
  • Later, the installed buildout is used to set up specific processes
    • Run as root in offline mode
    • Uses network configuration server
handout

Our philosophy is to separate software and configuration. We install software using RPMs. Later, we configure the use of the software using a centralized configuration database.

I'll briefly present the RPM building process below. This is interesting, in part, because it illustrates some interesting issues.

ZRS spec file (1/3)

%define python zpython
%define svn_url svn+ssh://svn.zope.com/repos/main/ZRS-buildout/trunk
requires: zpython
Name: zrs15
Version: 1.5.1
Release: 1
Summary: Zope Replication Service
URL: http://www.zope.com/products/zope_replication_services.html

Copyright: ZVSL
Vendor: Zope Corporation
Packager: Zope Corporation <sales@zope.com>
Buildroot: /tmp/buildroot
Prefix: /opt
Group: Applications/Database
AutoReqProv: no
handout

Most of the options above are pretty run of the mill.

We specify the Python that we're going to use as a dependency. We build our Python RPMs so we can control what's in them. System packagers tend to be too creative for us.

Normally, RPM installs files in their run-time locations at build time. This is undesirable in a number of ways. I used the rpm build-root mechanism to allow files to be build in a temporary tree.

Because the build location is different than the final install location, paths written by the buildout, such as egg paths in scripts are wrong. There are a couple of ways to deal with this:

  • I could try to adjust the paths at build time,
  • I could try to adjust the paths at install time.

Adjusting the paths at build time means that the install locations can;'t be controlled at install time. It would also add complexity to all recipes that deal with paths. Adjusting the paths at install time simply requires rerunning some of the recipes to generate the paths.

To reinforce the decision to allow paths to be specified at install time, we've made the RPM relocatable using the prefix option.

ZRS spec file (2/3)

%description
%{summary}

%build
rm -rf $RPM_BUILD_ROOT
mkdir $RPM_BUILD_ROOT
mkdir $RPM_BUILD_ROOT/opt
mkdir $RPM_BUILD_ROOT/etc
mkdir $RPM_BUILD_ROOT/etc/init.d
touch $RPM_BUILD_ROOT/etc/init.d/%{name}
svn export %{svn_url} $RPM_BUILD_ROOT/opt/%{name}
cd $RPM_BUILD_ROOT/opt/%{name}
%{python} bootstrap.py -Uc rpm.cfg
bin/buildout -Uc rpm.cfg buildout:installed= \
   bootstrap:recipe=zc.rebootstrap
handout

I'm not an RPM expert and RPM experts would probably cringe to see my spec file. RPM specifies a number of build steps that I've collapsed into one.

  • The first few lines set up build root.
  • We export the buildout into the build root.
  • We run the buildout
    • The -U option is used mainly to avoid using a shared eggs directory
    • The -c option is used to specify an RPM-specific buildout file that installs just software, including recipe eggs that will be needed after installation for configuration.
    • We suppress creation of an .installed.cfg file
    • We specify a recipe for a special bootstrap part. The bootstrap part is a script that will adjust the paths in the buildout script after installation of the rpm.

ZRS spec file (3/3)

%post
cd $RPM_INSTALL_PREFIX/%{name}
%{python} bin/bootstrap -Uc rpmpost.cfg
bin/buildout -Uc rpmpost.cfg \
   buildout:offline=true buildout:find-links= buildout:installed= \
   mercury:name=%{name} mercury:recipe=buildoutmercury
chmod -R -w .

%preun
cd $RPM_INSTALL_PREFIX/%{name}
chmod -R +w .
find . -name \*.pyc | xargs rm -f

%files
%attr(-, root, root) /opt/%{name}
%attr(744, root, root) /etc/init.d/%{name}
handout

We specify a post-installation script that:

  • Re-bootstraps the buildout using the special bootstrap script installed in the RPM.
  • Reruns the buildout:
    • Using a post-installation configuration that specified the parts whose paths need to be adjusted.
    • In offline mode because we don't want any network access or new software installed that isn't in the RPM.
    • Removing any find links. This is largely due to a specific detail of our configurations.
    • Suppressing the creation of .installed.cfg
    • Specifying information for installing a special script that reads our centralized configuration database to configure the application after the RPM is installed.

We have a pre-uninstall script that cleans up .pyc files.

We specify the files to be installed. This is just the buildout directory and a configuration script.

Repeatability and deployment issues

Repeating a buildout from a version of a configuraiton, and deployment issues to resolve

Repeatability

We want to be able to check certain configuration into svn that can be checked out and reproduced.

  • We let buildout tell what versions it picked for distributions

    • Run with -v

    • Look for outout lines of form:

      Picked: foo = 1.2

  • Include a versions section:

    [buildout]
...
versions = myversions

[myversions]
foo = 1.2
...

Deployment issues

  • Need a way to record the versions of eggs used.
  • Need a way to generate distributable buildouts that contain all of the source distributions needed to build on a target machine (e.g. source RPMs).
  • Need to be able to generate source distributions. We need a way of gathering the sources used by a buildout so they can be distributed with it.

PyPI availability

A fairly significant issue is the availability of PyPI. PyPI is sometimes not available for minutes or hours at a time. This can cause buildout to become unusable.

For more information

See http://www.python.org/pypi/zc.buildout