Article Cover by Lia Avant

Revision Manager

RevisionManager is a CVS frontend product for Zope. It brings the power of revision control to the ZODB, enabling different users to revise multiple objects simultaneously. The article gives some of the background for the development of RevisionManager and describes in detail how to install and implement it.

A UNIX classic: CVS

Whenever two or more people work together on a software project, revision control is a requirement for successful cooperation. It can also be useful for keeping track of changes in your software even when you are working by yourself.

Everybody who develops software has crossed the path of CVS, the Concurrent Versioning System. CVS has become the standard Open Source tool for revision management, although it is less than perfect, and even the Linux kernel source code is now managed with a commercial product, BitKeeper[2]. An Open Source successor to CVS, Subversion[3], is in sight, but currently far from stable.

As a Zope user or developer, you know that Zope is powerful in many ways. One of its strengths is that it gives you the possibility to delegate work. This delegation can take place not only inside Zope through the separation of content, representation and business logic, but outside it as well, where it is referred to as "collaboration". For example, you may have a team consisting of a designer for the HTML layout, a programmer to do the business logic and editors who write the content.

When working with Zope in a team, the same rules apply as with working on other development projects: project management, pair programming and unit testing are ingredients for a successful project, and so is revision control.

Zope already supports rudimentary revision control through Versions. Versions is a Zope product which makes use of the ZODB's history feature. Every time an object changes in the ZODB and that change is committed, the ZODB stores a new revision of that object. Old revisions are kept in the ZODB until the database is packed. You can view older revisions of objects over the Undo tab of most common Zope objects.

Zope versions have various advantages: They are a built-in Zope feature and, therefore, support almost all types of objects. They are also fully integrated into the management system (version and lock icons, management tab messages, etc.). They also allow the user to edit an object in-place without other people seeing the changes until version lock has been released.

The disadvantages are: While one person is editing objects in a Version they cannot be edited by other persons, since the objects are locked by Version. When the ZODB is packed, previous revisions are lost. Versions also does not permit the user to view the differences between two revisions of an object (there is no "diff" function, a standard in revision control).

So, why not use good old Unix technology?

Putting Zope and CVS together is not an entirely new idea. Steve Spicklemire's product
ZCVS Folder was the first to combine Zope and CVS. It remains somewhat popular in the community although it has never been assigned the status "stable" and the last release was in 2001.

Ariel Partners' CVSFile has a different underlying concept. It is an extended version of their ExternalFile product which adds CVS capability to it. As the name suggests, the actual data lies outside the ZODB in a file in the file system. This allows you to conveniently edit the files from the file system with your favourite editor without having to use ExternalEditor and such.

ExternalFile objects (and therefore CVSFile objects, as well) can behave like the most common Zope objects: DTMLDocument/Method, PageTemplate, File, and Image. The product cannot emulate PythonScripts. Furthermore, even though the ExternalFile objects can behave like these objects, they do not have their meta-type. While this does not matter so much when calling them directly, it does when you are trying to filter objects by meta-type or when you expect a certain behaviour from the object that CVSFile does not happen to implement. Finally, they do not behave like custom objects holding some sort of custom content and cannot be made to behave this way. The common denominator for all other Zope objects is "File" and that might not be enough for you.

The key problem in combining Zope and CVS around which most other details revolve is: CVS is a file system tool and Zope works with objects. While Spicklemire's ZCVSFolder (see above) tries to solve this problem by exporting objects in binary (zexp) or XML form, RevisionManager acknowledges that content objects primarily contain a single form of content which should be stored in its raw source on the file system. For example, it makes little sense to export an Image object as a Zope export file (zexp) when the image data is really what you want to keep track of.

RevisionManager supports all major CVS actions. These include importing, checking in and out, updating, adding and removing files, tagging and branching. The three common access methods are supported:

• local: The repository is on the same machine as the Zope server. On Unix, the user Zope is running under needs read/write permissions for the repository.
• server/ext: With this access method, CVS uses RSH or SSH to connect to a remote server. If SSH is used, public key authentication must be enabled and working because RevisionManager cannot enter a password into the SSH password prompt.
• pserver Using this access method, CVS connects to a server on port 2401 (pserver), authenticating with a user name and a password which needs to be entered into the RevisionManager configuration form.

• RevisionManager can use the Zope user id to connect to an RSH/SSH server
• Upon committing, RevisionManager can include the ZODB history of the objects in question in the CVS log.
• When enabled, RevisionManager stores properties of supported objects in separate

  .properties

  files.

In case you have not done this already, download the latest version of RevisionManager from the Zope.org homepage [6].

RevisionManager requires Zope 2.4 or later, the PageTemplates product and the CVS program. Zope 2.5 or later is recommended. Linux as the operating system is recommended, although the product should work on all common Unix versions. Windows is not officially supported, though RevisionManager is geared towards platform independence.

It is recommended that Zope run as a privileged user, for example as zope, not as nobody. In any case, it is absolutely necessary that the user Zope is running under have a valid home directory and have write access there. For example, when using the INSTANCE_HOME feature of Zope, you could use that as a home directory. Also keep in mind that all CVS transactions will be executed by the user Zope is running under, especially when using SSH (for the ~/.ssh directory) and public key authentication.

Like all other products, RevisionManager should be installed in the Products directory of your INSTANCE_HOME. When a RevisionManager instance is created, RevisionManager will create a directory named

var/RevisionManager

in your instance directory. Make sure it has the permission to create that directory and to write to it.

For each CVS module that you would like to use or create, you need an instance of RevisionManager. For example, if you have a Folder at /portal and you would like to manage all objects inside the portal folder, add RevisionManager to the portal folder. Thus, if you are serving many virtual hosts with one Zope instance and you wish separate CVS modules for them, you will have to create an instance in every folder that is serving a virtual host. For most users, one RevisionManager instance in the Root folder should work.

RevisionManager is instantiated just like any other Zope object. Simply select "RevisionManager“ from the "Add objects...“ tab in the appropriate folder. You will then be shown the add dialog. This dialog is divided into two sections, "Server settings“ and "Local settings“. Depending on the access method you want to use, filling out different fields is required:

• If you want to use a local repository on the same computer Zope is running on:
  1. Enter the appropriate repository path.
  2. Enter the module name. RevisionManager will fill in the ID of the folder it is being added to for the module name. It is a convention to give the module the same name as the root folder of the module.
  3. Specify a branch name if you want the initial checkout from a branch.
• If you are using the server access method:
  1. Enter the server address.
  2. Enter the appropriate repository path.
  3. Enter the module name. RevisionManager will fill in the ID of the folder it is being added to for the module name. It is a convention to give the module the same name as the root folder of the module.
  4. Specify a branch name if you want the initial checkout from a branch.
  5. Either enter a fixed user name for CVS to connect to the remote server or enable "Use Zope user id to connect to server“. CVS will then use the user id of the Zope user currently working with RevisionManager to connect to the server. This is useful if you want the CVS log to show the Zope user ids instead of the user name that Zope is running under. Note that password-less authentication must be enabled for this!
  6. Make sure the "RSH executable“ field points to the right program. If you want to use SSH, change this entry to match your SSH installation.
• If you want to use the pserver method:
  1. Enter the server address.
  2. Enter the appropriate repository path
  3. Enter the module name. RevisionManager will fill in the ID of the folder it is being added to for the module name. It is a convention to give the module the same name as the root folder of the module
  4. Specify a branch name if you want the initial checkout from a branch.
  5. Enter a user name and password to be used to connect to the pserver.

Depending on your installation, you may want to change the path to the CVS program in the "Local settings“ section of the form. If you are using the server or pserver access method, it might also be useful to increase the compression level for performance purposes. Enable "Save properties“ to make use of the property synchronization feature. If the debug feature is enabled, all output from the CVS program to stderr will be written to Zope's stderr, usually the console Zope was started from.

When you have completed filling in your settings, hit the "Add“ button. Depending on whether the module you specified already exists in the given repository or not, RevisionManager will show you one of two different forms.

If the module does not yet exist: You will be shown the import dialog. There, select the objects you want to be revision controlled. If you forget to include some objects, do not worry: you always have the possibility of adding items to or removing items from this list.

If the module already exists: RevisionManager will show a form with all files that are already available in the module which will enable you to map them to objects.

When something goes wrong during the process of adding an instance of RevisionManager, you may still find a RevisionManager instance in your folder which, however, has not been properly initialized. In general, it is best to delete it and start over from the beginning. Note that when you delete a RevisionManager instance, neither the CVS module nor the objects that were managed using it are deleted. At most, you will have to enter all your settings into the "Add" form again.

Though not a requirement, it is definitely an advantage to be comfortable with CVS in order to make use of RevisionManager's rich feature set. Because RevisionManager tries to perform as little magic as possible, you will be able to understand its behavior better when you understand the underlying CVS. If you are not yet familiar with CVS, I suggest reading the Cederqvist tutorial.

The Status tab is the control center for an instance. It not only gives you the most important information at first sight, you can also execute most CVS actions from here. The file tree shows all files in the module and their current status. The "Status“ column shows the object's CVS status. The "Working revision“ column shows the revision you are currently working with, whereas the "Repository revision" column shows the revision of the object that is in the repository.

If property synchronization is enabled, the "Property status“ and "Property revision“ columns will be shown as well. They contain the status of the property file for each object and the revision, respectively. A dash (-) indicates that no property adapter was found for that object.

All actions (executable by clicking one of the buttons) require at least one object to be selected. If you select a folder, the action will affect all objects and folders inside it. If you want the action to take effect on all files in the module, select the folder representing the root of the module.

Screenshot of the Status tab

If you are confused about something you see on one of the tabs in RevisionManager, you can click on the little "Help!“ link in the upper right-hand corner to access online help. Otherwise, handling RevisionManager should not be a problem for anyone familiar with CVS.

This section explains how RevisionManager was implemented and describes the hurdles that had to be taken during that process. It also explains where and how RevisionManager can be extended to fit custom needs.

Since CVS is a pure command line tool it has to be called as such. While this is not much of a problem, retrieving information from it is – its output has to be parsed. RevisionManager uses the CVSWrapper class which carries out all interaction with the CVS program. CVSWrapper exports the features of the CVS command line tool as a Python API so that for RevisionManager invoking CVS actions means calling a simple Python method. Further, CVSWrapper is not dependent on Zope and can be used in other Python programs that need to interact with CVS.

Instead of calling CVS directly, CVSWrapper actually uses ShellCommandProcessor to call the CVS program. ShellCommandProcessor is another Python class which allows calling a command line program platform-independently and makes it possible to read its output in a threaded environment such as Zope. Finding out that the output processing needed to be thread-aware took a lot of time. Without that thread-awareness, the Zope request transaction would simply lock up when CVS was invoked and returned large amounts of information that had to be parsed. Fortunately, help was found in ZCVSFolder, which uses the same thread-awareness trick.

Just like CVSWrapper, ShellCommandProcessor can be used in other Python programs independently of Zope.

As already mentioned, RevisionManager tries to retrieve the main contents of content objects. Unfortunately, there is no standard API for doing this. However, many content objects implement the FTP and WebDAV protocols, so RevisionManager literally abuses protocols: since FTP and WebDAV methods depend on a REQUEST, RevisionManager has to create fake response and request sets when using these methods, which is not without its problems. Moreover, File objects stream their data if they are larger than 64 KB, which makes special treatment for File and Image objects necessary.

The object contents are written to corresponding files on the file system, located in the var/RevisionManager directory of the Zope instance. Thus, before every CVS action, all objects have to be written to the file system; if the action modified the files on the file system (e.g. cvs update), the files will have to be rewritten back to the objects afterwards.

For optimization purposes, RevisionManager remembers which objects have been changed in the ZODB and on the file system at what time. This way, it only needs to synchronize those objects which have been modified since the last synchronization.

What if new files appear in the repository? They will have to be added to the ZODB as objects. But what metatype should be used for a file?

RevisionManager has a solution for this problem: it uses an object factory to create new objects with the file contents. Because the process of instantiating new objects is different depending on the object type, there can be no general factory to instantiate all types of objects. Instead, RevisionManager comes with factories for the most common Zope objects.

Furthermore, new factories for custom content objects can easily be added by writing a class that implements the

interfaces.ObjectFactories.IObjectFactory

interface and registers an instance of that class with

ObjectFactories.factoryRegistry.

There is more to an object than just the main contents. For example, the most common content objects and folders support setting arbitrary properties. ZSQLMethods do not only store their SQL code, they also store the ID of the database adapter to be used when executing their code plus half a dozen other settings (which essentially are properties). Since CVS operates with simple file system files, it has no ability to attach arbitrary data like properties to them.

As already mentioned above, RevisionManager can write property data to separate files, one extra property file per object. The files will be named

.id.property 

where id is the ID of the object holding the properties.

Because not all objects store their properties in the same way, RevisionManager allows one to implement new custom property adapters which adapt properties from your custom object to a form that RevisionManager can handle. You simply have to write a class that implements

interfaces.Properties.IPropertyAdapter

and register an instance with

Properties.propertyAdapterRegistry

RevisionManager already comes with property adapters for the most common Zope objects: for the ones extending OFS.PropertyManager (Folders, PageTemplates, DTML Documents, Files, Images) and for DTML Methods and SQL Methods.

When developing your application or filling objects with content using the Zope Management Interface (ZMI), you will probably find yourself having two browser windows open, one with the management screen for the object you are working on and one with RevisionManager to follow the objects' status in CVS.

In order to simplify the development process here, RevisionObjects was created as an add-on product for RevisionManager. It adds a new tab entitled "Versioning“ to the most common Zope objects (PageTemplate, DTML Document/Method, File, Image, PythonScript, ExternalMethod). From this tab you can execute nearly all CVS actions for that object, thus making it unnecessary to constantly keep the RevisionManager Status tab open in a separate browser window.

RevisionObjects may be downloaded from RevisionManager's Zope.org Homepage.

Because the number of objects in a CVS module managed with RevisionManager can be very high, it is impossible to display a list of all the objects at once. Not only would that require a lot of scrolling on the user's side, it also would take forever to render these lists. RevisionManager therefore uses trees to hide subitems from folderish objects, just like the Zope Management Interface (ZMI) does in its left-hand management tree.

RevisionManager's management pages (like the Status tab) are written as PageTemplates. While tree support for DTML is well-documented and widely-used, the contrary is the case for PageTemplates. ZTUtils.Tree is quite a complicated implementation of a cookie-based tree and the examples are unusable (i.e. they raise an exception) because they have not been updated in a long time.

Since we needed a tree implementation for ZPTs while the one available was unusable, ZopeTree was developed. It is light-weight, easy to use and independent of the templating language. The code is documented and it comes with interfaces, unit tests and two sample PageTemplates. Like RevisionManager, ZopeTree was released as Open Source and can be downloaded from the Zope.org homepage.

While RevisionManager already allows you easily to keep track of your software and content within Zope, there are still some issues with the current version of RevisionManager (as of this writing: 1.3):

• The current file system to ZODB synchronization is not very efficient. The files on the file system are only updated right before CVS is going to be run. This synchronization process consumes a lot of time. (If the files were always in sync with the objects, synchronization would not have to be performed every time before
• The CVS program itself is very inefficient. Its performance is proportional to the number of files in a module, not to the number of changes that are to be made. Subversion [3] does a much better job here.

The CVS Homepage:
http://www.cvshome.org

Commercial Revision Management System (Bitkeeper):
http://www.bitkeeper.com

An Open Source alternative to CVS (Subversion)
http://subversion.tigris.org

Zope CVS Mixin Classes Product by Steve Spicklemire:
http://www.zope.org/Members/sspickle/ZCVSMixin

CVSFile Zope Product by Ariel Partners:
http://www.zope.org/Members/arielpartners/CVSFile

Revision Manager Homepage:
http://www.zope.org/Members/philikon/RevisionManager

The CVS Manual:
http://www.cvshome.org/docs/manual

The Zope Tree product:
http://www.zope.org/Members/philikon/ZopeTree

Shane Hathaways Ape Product:
http://hathaway.freezope.org/Software/Ape