The first production version of Zope 3 was released on November 7. This was a major milestone in the Zope 3 project. Many people are probably wondering what Zope 3 means to them, if they should start using it, and why. In this article, I'll try to answer these questions or, at the least, to provide sufficient information to help people answer these questions for themselves.

First, I'll trace Zope 3's history, describing the reasons for decisions made along the way.

Zope was initially designed to empower non-technical users to easily manage information and create dynamic web applications. It achieves these goals in several ways, including providing:

• an environment that allows data, templates and scripts to be managed through-the-web,
• SQL methods and database connections that make it fairly easy to use relational data,
• an integrated indexing facility that makes it fairly easy to provide searching features,
• acquisition, which provides a simple way to share information in folder hierarchies; and
• ZClasses, which provide a basic way to create new content types and/or extend existing content types to provide new methods or templates.

These facilities worked well for lots of applications. In fact, there are some sophisticated content-management systems created by non-technical users, using just the through-the-web development facilities.

However, many developers need more than the through-the-web development facilities prebuilt objects provide. For these developers, a different set of development facilities and APIs are available. They can create additional file-system Python packages using a variety of APIs.

There are nonetheless some significant challenges for Zope 2 Python developers:

1. The frameworks are quite complex due in large part to the inherent complexity of the application domain. Zope provides many features, ranging from transaction-management and security to standard meta-data. New Zope software generally needs to work with all of these features. Inheritance, the mechanism used to manage complexity, doesn't scale well. To provide the most basic functionality a base class is provided, called “SimpleItem". The name suggests objects that don't do much else than provide basic functionality, while the base class itself has many base classes. Base classes are customized by overriding methods, or providing configuration data. Understanding what can be overridden or configured, and how the various base classes interact, can be quite difficult.
2. To provide rich functionality as easily as possible (especially for non-technical users) Zope does many things automatically. This works well for simple applications, but can get in the way of more sophisticated applications. Some prime examples of this are:
   • implicit acquisition; and
   • the namespace stack used by the Document Template Markup Language (DTML).
   With these facilities, a user can ask for data without having to spell out exactly where it should be found. This works well most of the time, but if there are unanticipated name conflicts, it can be difficult to tell where conflicting names come from. Even when things work, it can be a challenge to figure out why they work.
3. Zope provides a mechanism for easily installing third-party products. Unfortunately, it can be difficult to customize all these products. In traditional Zope products all of an object's functionality was provided by its class or base classes. The only way to change the behavior of products was to modify them or to subclass the classes they provide. Modifying products isn't generally practical, because it makes updating too difficult. Subclassing is unattractive because it creates overly tight coupling between the base classes and the subclasses. (This is known as the fragile base class problem.) One of the Zope Content-Management Framework’s (CMF) most important contributions (having nothing to do in this instance with content management) was a mechanism to provide presentation and application code in separate skins which could be managed independently from object classes, and could be readily customized. This allowed customization of presentation and application code without modification of the original source code and without having to resort to inheritance.

In late 2000, Zope Corporation engineers evolved ways for Zope to better serve developers. The main ideas to come out of these sessions were:

• Create a better templating system to make it easier for developers and web designers to collaborate. At the time, Zope's only templating language was DTML. DTML suffered from the problems most templating systems have, which is that the templates aren't valid web pages and can't be effectively updated with HTML editing tools. A designer might create a mock-up, but then lose control after a programmer made the moc-up dynamic. The result of this effort to create a better templating system was Zope Page Templates (ZPT), released in 2002.
• Provide a component model to make it easier to customize products i.e., to split product functionality into finer parts that could be independently updated.
• Provide cleaner separation of presentation and application logic through something like the traditional model-view-controller (MVC) or document-view frameworks.

Python is Zope's "secret weapon". Zope competes in a market dominated by Java-based solutions. Python is a major reason why developing applications for Zope is far more productive than developing for Java-based application servers. For all the reasons described above, however, Zope is not a very attractive development environment for Python programmers. Making Zope more attractive to Python programmers and allowing us to better leverage Python's strengths was a major goal for Zope 3 development.

Shortly after (and unrelated to) these brainstorming sessions, Zope Corporation hired the core Python development team, Python Labs, and they brought fresh perspectives and sensibilities.

As mentioned above, an early output of these brainstorming sessions was ZPT, which made it possible for developers and designers to collaborate on templates.

We realized that an MVC-like framework really needed a component architecture, so we investigated component architecture that would (among other features) support separation of the application and presentation logic.

Using a component architecture in a stateless environment such as HTTP was a major challenge. In a stateless environment the components needed for each request have to be assembled. This means that component assembly has to be easy.

Note that the CMF provides some of the advantages of a component system. Application code is taken out of content objects and distributed to tools and skins. Tools are components that provide specific application-support functionality. Skins provide separate presentation and application logic, with an emphasis on presentation. While the CMF provided some of what we needed and demonstrated the benefits of separating presentation and application logic from data management, we wanted to see if we could do better. In particular, we wanted to extend behavior using objects rather than using individual templates and scripts. The CMF's reliance on acquisition alone means that all templates and scripts share a common namespace regardless of content type. Names have to embed type information, and have to be carefully chosen.

We conceived an initial classification of components into content, view, and application components, loosely mirroring MVC, and designed a component-assembly mechanism using component registries and traversal components. Component assembly was based on connecting objects via interfaces.

We created an initial prototype that proved that the component-assembly ideas were workable. This was promising enough to motivate further development.

After this initial prototype, we evolved a more detailed story on how component developers worked, creating a short training course as a slide presentation that described the development process. This presentation was fictional, describing software that hadn't been written yet. The Python Labs team was instrumental in working out the initial development process and providing feedback. I gave the presentation many times, updating the course based on feedback. Essentially, this training course was the second prototype for the component architecture.

In creating the development process, a major goal (beyond support for component development) was to provide a development model that would be attractive to Python programmers. Zope 2 Python products have lots of very Zope-specific configuration information embedded in their Python code, which makes the Python code less approachable and detracts from understanding the code. An early decision was to separate configuration from application code. The Python code became simpler, and it became easier to use code that was not Zope-specific. The idea of separating configuration from application code turned out to be very powerful.

As this work was occurring, we realized we could make major improvements to the development experience by keeping an open mind about what developing for Zope could be like. We decided to approach the project as a major revision of Zope: Zope 3.

Near the close of 2001, we had iterated as far as we could with the tutorial and knew we needed to pursue a Zope 3 project with the following goals:

• Improve the development experience to provide a more rational development model, attractive to Python and other developers.
• Make it far easier to create objects usable in Zope:
  • 〈Make it possible to use existing Python classes (with little or no change).
  • Utilize features of the Zope framework with small incremental changes.
• Ease the leaning curve for developers
• Facilitate software reuse:
  • Use existing non-Zope-specific objects.
  • Add, remove, and replace the functionality of existing objects.
  • Change the user interface of existing objects.
  • Provide alternative access methods (e.g. FTP, XML-RPC, wxWindows) to objects.
• Provide more flexibility for configuring Zope software.
• Apply lessons leaned from using and building Zope 2.
• Integrate CMF technology and development patterns into Zope.

Tres Seaver had long advocated using Extreme Programming (XP) within Zope Corporation. We needed to bootstrap the Zope 3 implementation and I decided that we could try XP at the same time. We held a 3-day development session, which I called a "sprint" [Editor: see ZopeMag's miniGuide to Zope Sprinting to learn how sprints work], to create an initial minimal Zope 3 implementation as quickly as possible. We used pair-programming, stories, and unit tests. This initial session turned out to be extremely productive.

We held several other sprints at Zope Corporation, learning as we went along and making lots of important initial progress.

Zope 3 work began as a development branch of the Zope 2 tree. We quickly realized, however, that leveraging the existing Zope 2 code would make it hard to explore new ideas. Backward compatibility with the existing code was far too much of a distraction. We decided not to worry about backward compatibility initially and soon created a separate development tree for Zope 3.

In December 2001, we announced the Zope 3 project to the Zope Community as we were in the process of opening the Zope repository to contributors outside of Zope Corporation. Zope 3 provided a major opportunity for people to get involved in the Zope development process. We promoted the idea of sprints and held 3 Python sprints in Fredericksburg, in January and February of 2002. To date, there have been 28 community Zope 3 sprints. Sprinting has turned out to be great for:

• involving people in the Zope 3 development process,
• building the community; and
• completing important work.

The most important benefit of sprints is allowing people to meet, get to know each other, and collaborate in person. This makes it much easier subsequently to work remotely.

The Zope 3 project has also led to a huge increase in the number of people working on Zope itself. The majority of Zope code contributions and the vast majority of contributors come from outside of Zope Corporation. Zope 3 is truly a community project.

Three years passed from the initial Zope 3 project announcement through the initial release, which was much longer than had been anticipated. The vast majority of the features were developed in the first year, when the pace was very fast, building Zope 3 from a base of nothing to a system with a majority of the features in Zope 2, with contributions from a wide variety of contributors. At the end of 2002, we made what we thought would be the first alpha release.

In early 2003, we set about a series of "geddons", which were major restructurings intended to finish Zope 3 in time for a summer release. This was a time to evaluate what we'd done. Based on our experiences, however, we discovered aspects of the original designs that we wanted to change. By the fall of 2003, we had made a number of significant changes and had identified a number of changes still to be worked on that would take us well into 2004.

The period from early 2003 to mid 2004 was one of consolidation. Few new features were added. The focus was almost entirely on building a strong foundation upon which we could later build additional features. In most cases this represented major simplifications in the architecture. More code was removed that was later added.

Many of the changes made were significant. For example, we stopped using context wrappers to keep track of object location and switched to storing locations directly, leading to a vast simplification in location-aware code and object referencing. We completely rewrote a number of major subsystems. These changes greatly improved the architecture of the resulting system. They were possible because we were willing to take the time to learn from our initial mistakes and redo our work.

The decision to slow down and take our time, doubling the time to get to an initial release, was based both on a commitment to excellence and the realization that we didn't need to rush. Despite its shortcomings Zope 2 is still a great system and it has a long life ahead of it.

Over the course of work on Zope 3, our development processes evolved. We've:

• established a habit of testing,
• established a culture of excellence, learning to go slow to get things as right as we can, being willing to redo things when necessary; and
• created a new approach to testing that combines documentation and testing, which is rapidly becoming a positive habit.

By early 2004, there were several production projects using Zope 3. Clearly, Zope 3 was ready for production use even though some features were still missing or not ready to be finalized. Some people who could technically use Zope 3 in production couldn't risk doing so until the features they needed were stable. We decided to narrow the scope of the initial release to include just those essential features that were stable and could be used by people who don't need traditional through-the-web development features. We decided that Zope X3.0 would be a "developer's release", containing only those parts that were stable by early summer 2004.

A recent development is the Five project (http://codespeak.net/z3/five/), which provides facilities for using parts of Zope 3, particularly the Zope 3 component architecture., in Zope 2.

The current Zope 3 is not backward compatible with Zope 2, although, through the efforts of the Five Project, many parts of Zope 3 are usable in Zope 2.

The current Zope 3 is for Python developers. There is no stable through-the-web development support, although there are experimental facilities. I expect this to be the case for some time to come.

There aren't many third-party products for Zope 3 and parts that are mature in Zope 2, like cataloging, are still evolving in Zope 3.

With these limitations, Zope 3 provides some major advantages:

• It provides a much cleaner development model. *Developers* who've tried Zope 3 generally find it to be a much more productive environment than Zope 2.
• Zope 3 is much easier to adapt to meet specific application needs. There are interesting Zope 3 applications that don't look anything like traditional Zope applications. For example, there are applications that don't use Zope's traditional "object file” model. Zope 3 has proven to be far easier to use for relational-database applications.
• Zope 3 is designed from the ground up to support I18n and L10n software
• Zope 3 has good and improving documentation: a book (with more on the way), a tutorial, a built-in API reference, and a growing body of detailed internal developer documentation.

Zope 3 has both significant advantages and limitations relative to Zope 2. Whether it's right for you depends on your unique situation. Fortunately, you don't have to switch to Zope 3 anytime soon. Zope 2 will be with us for a long time. In fact, Zope 2 allows us to take our time with Zope 3. Thanks to the Five project, you can use parts of Zope 3 in your Zope 2 applications. Over time, Zope 2 will gain more Zope 3 features, easing the eventual transition to Zope 3.