Illustration by Brendan Davis

Practical Solutions Based on Popular Zope Products

This is the first article in a series about the author's practical experience in developing an advanced community site in Zope. The http://orl.cz site allows members to sign in and members can publish articles, ads, reviews, and newsitems. There is an advanced voting system, advanced search, sophisticated Webmail, and Website management for the site editor. A detailed description from the user perspective is available in English (and Czech) online at: http://www.spoxdesign.com/web/casestudies/orl_cz_details.

The articles will present interesting solutions achieved using proven Zope add-on applications such as Extensible UserFolder, Core Session Tracking (later integrated into Zope 2.5.1), LocalFS, IMP (a PHP-based Webmail system) integrated into Zope and PostgreSQL.

ORL.CZ needed an advanced login authentication system, featuring password recovery and integration with a PHP-based Webmail. While Zope has a rather sophisticated facility for managing users and a fine-grained permission system, authentication leaves a lot to be desired. To achieve the authentication system desired it was therefore necessary to use a special Zope product (with some minor adjustments).

People accessing pages protected by Zope itself will get a browser dialog based on HTTP basic authentication. Most users, however, prefer authentication without a browser dialog; the alternative, called "form-based authentication", involves presentation of a form within the Web page. Sadly, Zope has no form-based authentication out of the box. Several products have been developed to add this feature but, unfortunately, most of them have not been updated and therefore do not work with recent Zope versions. In my opinion the best option today is Extensible User Folder available from http://sourceforge.net/projects/exuserfolder/. Extensible User Folder, or XUF for short, allows the use of different sources for access to authentication information (username and password) and property information (such as user address, email address, etc.). Some of these sources may be used with vanilla XUF, such as PostgreSQL and ZODB. Others require a good deal of programming to be added to XUF. XUF works with recent Zope versions and has pretty decent documentation. Although version 1.0 is not even in the works, XUF releases so far have proved to work very reliably.

Since XUF is being developed at great speed, its API is prone to change without warning. Therefore, it might be prudent to use only the basic features of XUF and to develop simple add-ons in-house, so that your XUF application will work with various XUF versions. Once version 1.0 is released and the API is frozen, you could convert these add-ons to native XUF API calls.

If you use the basic HTTP authentication, central authentication comes automatically. Users defined in a User Folder (acl_users) placed in a Zope root folder are accessible everywhere. When using XUF, this is not so straightforward. Note, however, that developers should not remove the standard User Folder and replace it with an XUF instance in the site root. The developers of XUF explicitly warn against this: it is not only difficult to do properly but if done improperly could make all of Zope inaccessible for everyone including the Manager. In addition, even if done properly an upgrade of XUF could make Zope inaccessible.

The first obvious solution in this case is to place XUF instances one level higher, i.e. in every folder one level above the root. Each instance would use the same authentication source, for example PostgreSQL. While this seems to work, it is cumbersome. And it has practical drawbacks as well. First, you often need to display the name of the logged-in user in the root index_html, and if the XUF instance is not in the root, it would be necessary to store ns.SecurityGetUser().getUserName() in a session or use some other hack. This is tricky and error-prone. Second, any modification of the XUF instances would have to be copied into each subfolder, and also any XUF upgrade would require deleting the former instances, creating new instances in all the subfolders and then making any necessary modifications to all the new instances. Third, after trying this solution in real life I discovered that the re-authentication dialog appeared quite frequently when I was navigating between folders – separate XUF instances did not share the authentication token, which is correct but in this case undesirable.

According to the XUF FAQ, a root installation can be simulated using Virtual Host Monster, available in recent Zope versions such as 2.5.1 and 2.6.1. This is quite true. Here's how to do it:

• Make sure that you have a PostgreSQL table called passwd with at least four columns: username, password, roles,  password_plain. The last column is not needed for XUF itself, but it is used later on for Task 3: Webmail Integration.
• Create a subfolder, name it mysite, and move most of your root to mysite. Of course you should not move Control Panel, standard acl_users, browser_id_manager, session_data_manager, temp_folder and similar "system" items. Also make sure that you have a valid Database Connection instance to PostgreSQL in the mysite folder.
• Create a Virtual Host Monster (VHM) instance and in its Mappings screen define a mapping. If you have a local Zope installation, i.e. client and server on the same machine, this mapping would be "localhost/mysite". Beware that once you define this mapping you will not be able to access the true Zope root folder through http://localhost/manage, but you may still safely access it using an IP address http://127.0.0.1/manage. Also note that if Apache frontend with an appropriate rewrite rule is used, mapping is not necessary.

• Verify that when you access http://localhost/ Zope renders /mysite/index_html instead of /index_html.
• Create an XUF instance in the mysite folder. Specify the authentication source as PostgreSQL, and leave the default values for Table Name, Username Column, Password Column and Roles Column. Specify 'Null' as the properties source and 'Null' as the membership source, since we will manage user properties outside the XUF instance, although inside the passwd PostgreSQL table.

That's it. Your true root is now almost empty, the whole site has been moved to the mysite folder and a single XUF instance works transparently and centrally.

Please note that the Virtual Host Monster has some unresolved issues with ZCatalog. In short, a second, redundant ZCatalog entry may be created when you use reindex_object() for a DTML Document, ZClass or other instance. Even if it is not created, the value of absolute_url may be different after a reindex_object() call and after Update Catalog via ZMI. absolute_url sometimes includes the mysite subfolder, sometimes it does not. Current versions of Zope, including Zope 2.6.1, may therefore require a workaround, such as a small Python Script that computes the correct URL from the original absolute_url: .

url = context.absolute_url(relative=1)
urlFixed = url.replace('mysite/','')
return urlFixed 

Such a script may be then used as ZCatalog metadata.

Ninety percent of Web users cannot remember secure passwords. To prevent people from using simple, insecure passwords, current Websites generate secure passwords for users automatically and email them back. Many people will eventually forget or lose their passwords, which means that password recovery is an essential Website feature.

The owner of ORL.CZ required an uncommon solution: if a user forgot a password, they were to provide the secret answer to a question they had previously defined. If they were able to provide the answer, their password would be displayed on-screen. This requirement called for a plain (i.e. uncoded) password stored in an SQL table. Later on the owner decided to change to the standard method of password recovery, where the old password is not displayed but the user is requested to enter a new password. Nonetheless, the password_plain column in the SQL table (see page 2) proved to be indispensable for Task 3: Webmail Integration.

Instead of using the XUF API to add users, I decided to create independent Python Scripts (and ZSQL Methods, of course) which insert the data directly into the SQL table used by XUF. This solution provides full control of new user creation.

In the following code example, SDM is an instance of Session Data Manager.

Values from a Web page form are stored in getSessionData()['form']. A two- line Python Script formdatasave with a single parameter form_data provides a universal way to save form data:

data=context.SDM.getSessionData()
data.set('form',form_data) 

This script is called from a DTML Method as follows:

<dtml-call "formdatasave(REQUEST.form)"> 

Later on, the saved data will be retrieved from the session object, and the password will be encrypted:

data=context.SDM.getSessionData()['form']
new_password=context.encrypt(data.get('password'),new_username)
new_password_plain=data.get('password') 

The encryption used here in context.encrypt must be the same as the encryption expected by XUF, since we will save this data in the SQL table passwd which is used by XUF. Since Python Scripts do not allow one to import the necessary crypt module, the encryption takes place in a short External Method encrypt. Here is the source code of encrypt:

def encrypt(pwd,salt):
   from crypt import crypt
   encpwd=crypt(pwd,salt)
   return encpwd

You can see what the XUF requirement for correct encryption is: the salt parameter must equal the username.

When the user requests the change of a forgotten password, you can simply prompt them for the secret answer to their question, and use a ZSQL method to verify it:

<dtml-if "_.len(get_userid_by_answer
(username=username,answer=answer))">
  ... enter code to change password ...
<dtml-else>
  <font color=red>Incorrect secret answer.</font>
</dtml-if> 

The get_userid_by_answer ZSQL Method is very simple:

SELECT userid FROM passwd
WHERE <dtml-sqltest username op=eq type=string>
  AND <dtml-sqltest answer op=eq type=string>; 

This password recovery mechanism should be easily accessible. Indeed, a user should be immediately and automatically presented with the recover password option if the originally entered password is incorrect. If you examine the docLogin DTML method inside the XUF instance, you'll see the <dtml-if authFailedCode> condition near the top. This condition is ideal for placement of our recovery reminder. However, if one wants this to be displayed somewhere else on the screen, the following code can be added to docLogin:

<dtml-if authFailedCode>
  <P>Forgotten your password? 
Perhaps you remember the secret answer to a 
<A HREF="/recover_password_1"> question</A>.</P>
</dtml-if> 

ORL.CZ uses the sophisticated and popular Open Source Webmail system IMP, which has its own authentication facilities. Integration into ORL.CZ required some means of bypassing this authentication since we already had user authentication for the Website. To achieve this in a secure manner, the XUF authentication data in the SQL table was securely passed to a slightly modified IMP authentication module. That is why the SQL table passwd used by XUF contains the password_plain column. Needless to say, the SQL table containing the plaintext passwords must be well secured.

Since IMP Webmail is an interface to an IMAP or a POP server, it expects an existing user account accessible on a local or remote server. To streamline user experience, this account is created in Zope when a new user is registered simply by calling an external method (or a cgi script provided by a Web hosting company if you use hosting services for IMAP and/or Zope). And this is one of the reasons why the authentication data is entered by custom Python Scripts directly into the SQL table passwd.

IMP integration is described in detail in the following section.

IMP, available from http://www.horde.org, is part of a comprehensive Horde framework. IMP itself provides a full-featured Webmail, while other Horde components include a calendar, a file manager, an NNTP client, an address book and other goodies. IMP is written in the Web scripting language PHP. There is simply no equivalent Zope Product available.

I was completely new to PHP when I started trying to integrate Zope and IMP. After a few days of reading PHP documentation, studying Horde framework source code and a lot of trial and error I learned how to modify the Horde/IMP source code and achieve passwordless authentication. Originally tested with Zope 2.4.3 and IMP 3.0, this solution now works with Zope 2.5.1 and should be OK for all future Zope 2.x. You will also need an IMAP server or a POP3 server (this is an IMP requirement). IMP is simply an interface to an IMAP or a POP server, not a standalone complete e-mail system.

My goal was to allow any user currently authenticated in Zope on the ORL.CZ site to access IMP. The user would not need to re-authenticate manually to access IMP. I wanted to achieve this with as little modification of IMP source code as possible, so that future upgrades of IMP would be relatively painless and would not require extensive patching. Simply put, if a Zope Web page knows that a user is authenticated, it must assure the IMP authentication page that this is indeed the case, thereby bypassing IMP authentication.

As mentioned above, IMP is a part of Horde, and normally Horde does the authentication business and then allows access to various Horde applications, such as IMP. Since in our case we only needed IMP, Horde had to be reconfigured to let IMP handle the authentication task. A brief look at the Horde source tree reveals that this authentication is processed in imp/redirect.php. The imp/redirect.php file expects two variables POSTed from the preceding Web page: imapuser and pass.

Why imapuser? Again, IMP is just an interface to an IMAP or a POP3 server. This also means that a POP/IMAP user account will have to be created by Zope long before Zope passes authentication data to IMP, as mentioned in the previous section.

The following four integration methods are possible:

1. Write a form with hidden fields imapuser and pass in the Zope page, and submit the form via POST to the IMP authentication page imp/redirect.php. This has the security drawback of being visible in the HTML source code of the Zope page, and being saved in Web caches.
2. Only provide imapuser via POST and allow IMP itself to access the passwd PostgreSQL table directly to retrieve the plain password from the password_plain column. This would be terribly insecure, because anyone with a decent knowledge of forging Web requests could ask for any password by simply providing a username.
3. In a Zope page, set a browser cookie that contains the encrypted user name and password, and have imp/redirect.php pick up that cookie. Encryption can be cracked but this it certainly looks more secure than methods 1 and 2.
4. Use some form of communication between a Zope page and the PHP authentication page, probably through SQL.

Since the fourth approach seemed to me to be the most secure one, I decided to go for it. The Zope page would create a token or tag and insert a record containing this tag, the username and the password into a special SQL table. This SQL table could be accessed by IMP. The tag, which acts as a key to retrieve SQL data, would be stored in the client browser cookie. IMP would select the record using the tag retrieved from the cookie, and then delete the record.

There are both simple and sophisticated methods for creating a random tag. The easiest approach is simply to use remote address and the exact time, as in the following Python Script set_cookie

Python Script set_cookie:

request = container.REQUEST
RESPONSE =  request.RESPONSE
luser=ns.SecurityGetUser().getUserName()
password_plain=context.get_pwd(username=luser)[0].password_plain
tag1=request.REMOTE_ADDR
tag2=DateTime().timeTime()
tag=tag1+"-"+repr(tag2)
context.insert_tag(tag=tag,username=luser,password=password_plain)
RESPONSE.setCookie('zopeimp',tag,path='/')
return 

set_cookie above is called by the short Python Script open_webmail below. And since open_webmail redirects and does not display anything, it may be used as a Web page anchor: <A HREF=”open_webmail”>Open Webmail</A>

The Python Script open_webmail:

Click Here for the code example

NOTE: In most installations Zope is installed behind Apache. This is a suitable configuration for the IMP integration. Two simple rewrite rules in the Apache config will send Zope requests to a Zope server and IMP requests to the PHP source code. Apache will serve everything from port 80. This is important, because if Zope was using a different port from IMP (such as port 8080), IMP would not be able to get a cookie from an ":8080" host.

set_cookie must also insert the tag, username and password into a special zopeimp SQL table. This is carried out by the ZSQL Method insert_tag (the parameters are tag, username, password):

insert into zopeimp values(
  <dtml-sqlvar tag type=string>,
  <dtml-sqlvar username type=string>,
  <dtml-sqlvar password type=string>); 

zopeimp SQL table definition:

taq             text
username        text
password        text 

As described earlier, imp/redirect.php needs some modifications. Actually, it would suffice to copy imp/redirect.php to a different filename, such as imp/zoperedirect.php, and modify it. The original redirect.php may be left where it is, because it will not interfere with our integration and it will allow a separate login to IMP without Zope if desired.

The patch of imp/redirect.php does not modify the existing source code. Instead, it comprises a few lines of code inserted into the source. I will present the patch step by step:

if (!isset($HTTP_COOKIE_VARS["zopeimp"])) {
   echo "<h2>Your browser did not send the necessary cookie</h2>";

Some error checking will not hurt. This should catch browsers with disabled cookies.

} else {
   $zi_cookie=$HTTP_COOKIE_VARS["zopeimp"];
   $zi_tag=stripslashes($zi_cookie);

Zope quotes cookies automatically. stripslashes() is a PHP string function that un-quotes the zopeimp cookie.

   $zi_tag=substr($zi_tag,1,strlen($zi_tag)-2);
substr removes the original quotation marks... 

   $zi_tag="'" . $zi_tag . "'";

... and single quotes will now adorn the $zi_tag so that it may be inserted into the following SQL SELECT statement.

   $zi_c=pg_Connect("host=localhost port=5432 dbname=maindb 
user=john password=secret");

Note that user john should be able to access the zopeimp table, but not the passwd table.

Now we can run two queries to retrieve a data row and immediately delete it:

   $zi_r=pg_exec($zi_c,"SELECT username,password FROM zopeimp WHERE
tag=$zi_tag;");
   $zi_r2=pg_exec($zi_c,"DELETE FROM zopeimp WHERE tag=$zi_tag;");

Another error check: pg_numrows() should contain precisely one record.

   $zi_num=pg_numrows($zi_r);
   if ($zi_num==1) {
      $HTTP_POST_VARS['imapuser']=pg_result($zi_r,0,"username");
      $HTTP_POST_VARS['pass']=pg_result($zi_r,0,"password");
      // BE SURE TO CUSTOMIZE POST VARS TO YOUR NEEDS!
      $HTTP_POST_VARS['server'] = 'localhost';
      $HTTP_POST_VARS['actionID'] = '105';
      $HTTP_POST_VARS['mailbox'] = 'INBOX'; 
      $HTTP_POST_VARS['port'] = '143';
      $HTTP_POST_VARS['maildomain'] = 'domain.com';
      $HTTP_POST_VARS['protocol'] = 'imap';
      $HTTP_POST_VARS['realm'] = 'domain.com';
      $HTTP_POST_VARS['folders'] = 'mail%2F';
      $HTTP_POST_VARS['new_lang'] = 'cz_CZ';
      $HTTP_POST_VARS['button'] = 'P%F8ihl%E1%B9en%ED+do+syst%E9mu';
   } else {
      if ($zi_num==0) 
      {
         echo "<h2>Your browser sent a cookie we do not have on
file!</h2>"; 
      } 
   }
} 

The whole patch presented above should be placed in a copy of imp/redirect.php between

define('IMP_BASE', dirname(__FILE__));
  require_once IMP_BASE . '/lib/base.php'; 

and:

  $action = Horde::getFormData('action', ''); 

open_webmail
	@set_cookie
		@ get_pwd
		@  insert_tag into zopeimp
		@ save cookie in the client browser
	@ redirect

set_cookie reads the plaintext password of the current user. Then set_cookie invents a pretty unique tag, and inserts a new record into a special zopeimp table. The record contains the tag, the username and the plaintext password. Since the tag will be used by IMP to retrieve the password, set_cookie stores the tag in a cookie. This cookie expires at the end of a browser session. A marginally cleaner solution would be if IMP forced cookie expiration.

The record in the zopeimp table is destroyed automatically by IMP. It exists for a split second, acting as a momentary conduit between Zope and IMP. And that's the end of set_cookie. We are back at open_webmail. Now open_webmail redirects to: http://www.mysite.com/horde/imp/zoperedirect.php.

zoperedirect.php picks the tag from the cookie. It removes the double quotes and replaces them with single quotes, because PostgreSQL does not like double quotes in a parameter value. zoperedirect.php now uses the tag to retrieve the username and the password, immediately destroying the record that it just retrieved. Then it fills in the remaining values that are necessary for successful IMP sign-in.

Is this solution really secure? Well, yes and no. The weak spot is the get_pwd ZSQL Method. If someone knows about this method, and knows a username, he would be able to retrieve a password for the username simply by calling get_pwd. Since get_pwd does not appear in any HTML source, this solution is fairly secure - as long as the attacker does not know the name of the get_pwd method. In the case of Zope procedure names spilling onscreen – for example, after a cracker's deliberate entry of an invalid URL – you could be in trouble. Further, if some of the original developers who know the method name defect to the competition or leave your company, they could retrieve passwords. To prevent this you can and should set permissions for the get_pwd ZSQL Method so that it is accessible only to the role 'Authenticated'. Any cracker would then have to have a valid account in your system, and his actions would be recorded in the Undo logs.

ZSQL Methods are, therefore, somewhat weak from the security standpoint, especially for large deployment. A better solution would be to store the plain password elsewhere. A good place for it would be a cookie session. A modified XUF product would be able to store a plain password in a session on the server, but this is beyond the scope of the present article. Given that XUF is a fast moving target, I settled for the solution described since it works reliably and the number of ORL.CZ users is not very large.

An attacker sitting somewhere in-between the browser and the server might intercept the open_webmail request, capture the tag, cut the line to the server so that the browser would not receive "redirect.php" from the server, and present themselves as the browser using the tag.

However, any attacker sitting between browser and server does not need to go to such lengths in order to read a user's mail. He can simply retrieve the password during form authentication, because HTTP is unencrypted. In other words, there are more serious issues for this "in-between" person than cookie hijacking. There is only one way to stop these activities: use SSL.

There are other methods for achieving the above-described results. One method, I thought, would be to use the Zope product PHPDocument. This product is supposed to make it possible to run PHP code from within Zope. Such a solution would be much simpler than the one presented in this article, because all the cookie and SQL magic would be unnecessary. Unfortunately, I tried this solution but PHPDocument and its successors did not work properly with IMP. Another method would be to keep the plaintext password in a session, such as CoreSessionTracking . The problem here is that you would need a hook in XUF to execute a "save to cookie" method on successful login. It would be great if XUF included a hook to have a user-defined method run when a successful login occurs, but this is not currently the case.

Milos Prudek: Milos Prudek has been working professionally in the IT industry since 1988. While working as an application programmer with FoxPro and Pascal, he also taught a wide range of IT subjects to different audiences. A freelance developer and writer since 1991, he established a local ISP company, which he has since expanded. He has been active in Zope development since 1998 and runs a small Web usability and Web development company Spox Design. His only passion is promoting tolerance among people.