Pinned versions

During the Neanderthal Grok Sprint, Luciano and I have been working on "pinning down versions" to protect Grok development and deployment from broken, incompatible and conflicting egg releases of all the dependencies.

The first thing we did is to come up with and test (on Linux, OS X and Windows) a list of known working egg versions. You can see the current list here.

With this list of versions, we updated Grok's buildout.cfg to:

  1. use the versions.cfg by way of the extends directive.
  2. point the versions directive to the versions section that is defined in the versions.cfg file.

What this now means for the development of Grok and the development with Grok is described below.

NOTE: Now that we have a set of versions that actually work, we can evolve this to something with as much final releases in at as possible.

For developers of Grok

The buildout.cfg of Grok now use the versions and extends directive like so:

[buildout]
....
extends = versions.cfg
versions = versions # being defined in the versions.cfg file
....

Now, when running ./bin/buildout it will make sure exactly those versions of the egg dependencies are downloaded and installed for this Grok buildout.

During Grok's lifecycle newer versions of the egg dependencies might be released. If you want to test such a newer version, you could temporarily modify the buildout.cfg and override the versions section like so:

[buildout]
extends = versions.cfg
versions = versions # being defined in the versions.cfg file
....

[versions]
# whatever specific egg version is defined in this section,
# will override the version in the versions.cfg
zope.i18nmessageid = 3.5.0

One other way to do this is to create a temmporary config file, say dev.cfg and experiment with newer eggs versions like so:

[buildout]
extends = buildout.cfg

[versions]
# whatever specific egg version is defined in this section,
# will override the version in the versions.cfg
zope.i18nmessageid = 3.5.0

Make sure you use this config file when running the buildout command like so:

$ ./bin/buildout -c dev.cfg

If the newer egg version proves to be compatible, the newer version number can be put in the versions.cfg and checked in. We think it is very important to make sure the newer egg version and its dependencies are to be available on at least Windows too.

Whenever a Grok release tag is made, we obviously have a tagged fixed versions.cfg file as well. Besides the regular steps needed to make an actual Grok release, extra steps will be required to support grokproject (the reason is explained in the following section):

  1. the tagged versions.cfg needs to be uploaded to http://grok.zope.org/releaseinfo/grok-[versionnumber].cfg
  2. the version of the Grok egg needs to be added to the grok-[versionnumber].cfg file, e.g. grok = 0.11.
  3. the contents of http://grok.zope.org/releaseinfo/current needs to be updated to contain the name of the current 'released' Grok versions info file. E.g. grok-0.11.cfg.

For developers with Grok

We think it is reasonable to assume that users of Grok, in other words application developers that use Grok as a basis, are using grokproject for setting up the initial project structure they'll use. At least they should be strongly encouraged to do so.

We should also support upgrading an already existing project that has been created with grokproject, i.e. when a new Grok has been released.

Initial project creation

For the grokproject user, nothing will change really. He will issue the grokproject command and this will setup an project structure for him.

What the new grokproject will do by default is this:

  1. fetch the name of the current versions file from http://grok.zope.org/releaseinfo/current
  2. add to the buildout.cfg an extends directive pointing to the URL of the named versions file. E.g. http://grok.zope.org/releaseinfo/grok-0.11.cfg
  3. add the versions directive pointing to the [versions] section in the named versions file

Now when the buildout command is run, exactly those versions of the eggs will get installed.

When running grokproject this behaviour can be overridden. The application developer can choose to specify the URL pointing to custom versions file like so:

$ grokproject \
  --version-info-url=http://hostname/some/path/versionsfile.cfg

The extends directive will then point to that URL.

NOTE: the implementation for the "new" grokproject lives on the "jw-luciano-use-versions-file" branch until it is merged, probably somewhere next week.

Overriding versions of particular eggs

It is likely that the application developer introduces more dependencies on other software. He should of course add this dependecy in his setup.py file. If he wants to make sure that a specific version of the dependency is used when deploying the project, he can add this specification to his buildout.cfg like so:

[buildout]
extends = http://grok.zope.org/releaseinfo/grok-0.11.cfg
versions = versions
....

[versions]
spam = 1.2.3

Likewise, if the application developers wants to override the version of one of the eggs grok depends on he can adjust his buildout.cfg to look like this:

[buildout]
extends = http://grok.zope.org/releaseinfo/grok-0.11.cfg
versions = versions
....

[versions]
zope.i18nmessageid = 3.5.0 # will override the original spec

By the way, users of grokproject should be encouraged to let the Grok developers know about wanting to use a newer version of a current Grok dependency. With this information, the Grok developers can consider this newer version to be specified - "blessed" - for the upcomming release of Grok.

Upgrading to a new Grok release

Whenever an application developer is ready to upgrade his project to a new version of Grok, he only needs to change the URL that points to the versions file to the newest versions file:

[buildout]
extends = http://grok.zope.org/releaseinfo/grok-0.99.cfg
versions = versions
....

Then, he needs to run his buildout command again to make it install the eggs that are now specified for this latest release of Grok.

"Fixing up" current grokproject-made projects

Projects that have been created with the current grokproject command can be "fixed up" by:

  1. removing the newest = false line from the project's buildout.cfg.

  2. adding the following two lines to the [buildout] section:

    [buildout]
    ....
    extends = http://grok.zope.org/releaseinfo/grok-0.10.1.cfg
    versions = versions
    

and then running the buildout command again.

NOTE: this "fix up method" will work when Grok 0.10.1 has been released. If you want to try anyway, you could try using this URL: http://grok.zope.org/releaseinfo/grok-0.10.1b1.cfg