Personal tools
You are here: Home → Documentation → Tutorials → Introduction to zc.buildout → Source, Binary and RPM experiments
 

Source, Binary and RPM experiments

Distributing your buildout as source or binary, experiments with the RPM format
Jim Fulton's tutorial for using buildout, originally given at DZUG 2007
Page 16 of 17.

Source vs Binary

  • Binary distributions are Python version and often platform specific

  • Platform-dependent distribution can reflect build-time setting not reflected in egg specification.

    • Unicode size
    • Library names and locations

  • Source distributions are more flexible

  • Binary eggs can go rotten when system libraries are upgraded

    handout

    Recently, I had to manually remove eggs from my shared eggs directory. I had installed an operating system upgrade that caused the names of open-ssl library files to change. Eggs build against the old libraries no-longer functioned.

RPM experiments

Initial work creating RPMs for deployment in our hosting environment:

  • Separation of software and configuration
  • Buildout used to create rpm containing software
  • Later, the installed buildout is used to set up specific processes
    • Run as root in offline mode
    • Uses network configuration server
handout

Our philosophy is to separate software and configuration. We install software using RPMs. Later, we configure the use of the software using a centralized configuration database.

I'll briefly present the RPM building process below. This is interesting, in part, because it illustrates some interesting issues.

ZRS spec file (1/3)

%define python zpython
%define svn_url svn+ssh://svn.zope.com/repos/main/ZRS-buildout/trunk
requires: zpython
Name: zrs15
Version: 1.5.1
Release: 1
Summary: Zope Replication Service
URL: http://www.zope.com/products/zope_replication_services.html

Copyright: ZVSL
Vendor: Zope Corporation
Packager: Zope Corporation <sales@zope.com>
Buildroot: /tmp/buildroot
Prefix: /opt
Group: Applications/Database
AutoReqProv: no
handout

Most of the options above are pretty run of the mill.

We specify the Python that we're going to use as a dependency. We build our Python RPMs so we can control what's in them. System packagers tend to be too creative for us.

Normally, RPM installs files in their run-time locations at build time. This is undesirable in a number of ways. I used the rpm build-root mechanism to allow files to be build in a temporary tree.

Because the build location is different than the final install location, paths written by the buildout, such as egg paths in scripts are wrong. There are a couple of ways to deal with this:

  • I could try to adjust the paths at build time,
  • I could try to adjust the paths at install time.

Adjusting the paths at build time means that the install locations can;'t be controlled at install time. It would also add complexity to all recipes that deal with paths. Adjusting the paths at install time simply requires rerunning some of the recipes to generate the paths.

To reinforce the decision to allow paths to be specified at install time, we've made the RPM relocatable using the prefix option.

ZRS spec file (2/3)

%description
%{summary}

%build
rm -rf $RPM_BUILD_ROOT
mkdir $RPM_BUILD_ROOT
mkdir $RPM_BUILD_ROOT/opt
mkdir $RPM_BUILD_ROOT/etc
mkdir $RPM_BUILD_ROOT/etc/init.d
touch $RPM_BUILD_ROOT/etc/init.d/%{name}
svn export %{svn_url} $RPM_BUILD_ROOT/opt/%{name}
cd $RPM_BUILD_ROOT/opt/%{name}
%{python} bootstrap.py -Uc rpm.cfg
bin/buildout -Uc rpm.cfg buildout:installed= \
   bootstrap:recipe=zc.rebootstrap
handout

I'm not an RPM expert and RPM experts would probably cringe to see my spec file. RPM specifies a number of build steps that I've collapsed into one.

  • The first few lines set up build root.
  • We export the buildout into the build root.
  • We run the buildout
    • The -U option is used mainly to avoid using a shared eggs directory
    • The -c option is used to specify an RPM-specific buildout file that installs just software, including recipe eggs that will be needed after installation for configuration.
    • We suppress creation of an .installed.cfg file
    • We specify a recipe for a special bootstrap part. The bootstrap part is a script that will adjust the paths in the buildout script after installation of the rpm.

ZRS spec file (3/3)

%post
cd $RPM_INSTALL_PREFIX/%{name}
%{python} bin/bootstrap -Uc rpmpost.cfg
bin/buildout -Uc rpmpost.cfg \
   buildout:offline=true buildout:find-links= buildout:installed= \
   mercury:name=%{name} mercury:recipe=buildoutmercury
chmod -R -w .

%preun
cd $RPM_INSTALL_PREFIX/%{name}
chmod -R +w .
find . -name \*.pyc | xargs rm -f

%files
%attr(-, root, root) /opt/%{name}
%attr(744, root, root) /etc/init.d/%{name}
handout

We specify a post-installation script that:

  • Re-bootstraps the buildout using the special bootstrap script installed in the RPM.
  • Reruns the buildout:
    • Using a post-installation configuration that specified the parts whose paths need to be adjusted.
    • In offline mode because we don't want any network access or new software installed that isn't in the RPM.
    • Removing any find links. This is largely due to a specific detail of our configurations.
    • Suppressing the creation of .installed.cfg
    • Specifying information for installing a special script that reads our centralized configuration database to configure the application after the RPM is installed.

We have a pre-uninstall script that cleans up .pyc files.

We specify the files to be installed. This is just the buildout directory and a configuration script.

Next: Repeatability and deployment issues Previous: zc.sharing example