SubscribeXMLTagsEditHistoryDiscussion

  1. Introduction
  2. Status
  3. Goals
  4. Scenarios
  5. Motivations
    1. What about ATOM feeds?
  6. Internal design
    1. svnwiki-mail program
    2. Format for discussion pages
    3. SQLite tables
      1. Email addresses
      2. Subscriptions
      3. Mailing lists
      4. Global variables
  7. Managing subscriptions
    1. Subscribing to a wiki page
      1. Confirmed addresses
      2. Confirmation required
    2. Confirmation
      1. Invalid password
      2. Successful subscription
    3. Unsubscribing
    4. Unsubscribing from all pages
  8. Notifications
    1. Modifications to pages
    2. Discussion posts
  9. Mail to discussion gateway
  10. Tasks
    1. Done
    2. Pending

Introduction

This document specifies the design for the email extension for Svnwiki.

Status

The specification is complete. We are in the process of implementing it.

Goals

The email extension makes wiki pages and directories of wiki pages behave like mailing lists with the goals of:

Scenarios

The following are some scenarios that we expect to be enabled by the email extension:

Motivations

The mail user agent interfaces can have a huge impact on the way people use email, on things like

This can be greatly seen by contrasting the usage of email by different organizations with different email systems. I recently switched from working at Novell to Google, both organizations that run their email on systems they wrote and sell to other organizations (GroupWise and Gmail). These organizations have very different email use patterns, many of which can be traced to characteristics of their systems' user interfaces. Users are gradually adjusting their practices to match the features offered by their email systems.

Mainstream mail user agent interfaces have improved significantly in recent times in how convenient they make it for most people to handle large volumes of messages. Perhaps the most important features in this regard are

The number of messages that mainstream mail user interfaces encourage users to send and receive is increasing significantly.

Another important change for mainstream email users is the availability of practically unlimited disk space for storing text-only emails. While the cost of disk space has always been decreasing, only recently did it get to the point where deleting messages (other than spam) is more expensive (in terms of time invested and the cost of not having previous information available) than storing them for perpetuity for most email users.

These two factors, the improvements in the user interfaces and the advent of unlimited space for storing email, conspire to make email more useful than it was before. The Svnwiki mail extension aims to leverage these improvements by creating a mailing list for every wiki page and making it convenient for users to subscribe to them. We believe that most Svnwiki users will subscribe to lots of different pages (mailing lists) and start receiving notifications about events that interest them and which they would otherwise miss.

We expect most pages to behave as relatively low traffic mailing lists. As such, our interface encourages users to subscribe to many different pages, each of which will send messages very sporadically.

What about ATOM feeds?

Svnwiki already supports building an ATOM feed for every wiki page. While some of the functionality provided by the mail extension is redundant with the ATOM feeds—a user can already subscribe to the feeds and receive most of the information he will receive through the email extension—, we believe that the email extension will provide additional value.

From a user's perspective, ATOM feeds may have an advantage over subscribing to pages to receive email notifications: subscribing to pages requires trusting the servers' administrators

The first concern is not as important as it was in the past as the accuracy of spam filtering technologies increases. As for the second, its relevance is also diminishing, this time by the availability of filters as well as practically unlimited disk space and bandwidth (meaning that, most of the time, being unable to unsubscribe from a mailing list is just a minor inconvenience). Nevertheless, users subscribing to an ATOM feed exert more direct control over the infrastructure involved. To somewhat mitigate this, we will be very careful to make it very easy for users to unsubscribe from a wiki page in very little time.

That said, the value provided by the email extension will likely surpass the value of the ATOM feeds provided by Svnwiki. By a large margin, too:

Users who prefer to receive notifications through ATOM feeds instead of email will still be able to do so.

Internal design

svnwiki-mail program

The mail extension will come with an optional svnwiki-mail program. The purpose of this program is to allow users to

directly from their email program. The administrator should set his mail delivery agent so all messages sent to an address such as wiki+X@freaks-unidos.net, for all reasonable values of X and then set the svnwiki:mail:gateway-address property in the wiki to the address resulting of using the empty string as X (wiki+@freaks-unidos.net in this case). X will be used to pass parameters to the svnwiki-mail program.

The usage of the svnwiki-mail program is optional. The mail extension will assume its availability only if the svnwiki:mail:gateway-address property is set.

More details about this program are provided below.

Format for discussion pages

Currently, Svnwiki uses an adhoc format for storing discussions where comments for file foo/bar are stored one after the other in foo/xsvnwiki-discuss/bar, with some simple separator. This has worked well so far and has the advantage of requiring little additional logic for producing the views for the discussions. However, it is beginning to get out of hand for large discussions such as:

One of the things that the mail extension will do is allow users to have big conversations over email and add their messages to an archive where others can read them. It makes little sense to keep two separate forms of conversation, one for the web-based discussions and another for email threads. For this reason, the representation of discussions will have to be modified to a form that makes it more useful to generate views for big archives of mail / comments.

For a page foo/bar, all its associated messages will be kept in separate files in a foo/xsvnwiki-discuss/bar directory. The names of the files will be of the form 1196649287:0, where the first number is the timestamp of the message's date and the second number is a unique ID within that timestamp (it's only purpose is to allow us to have multiple messages with the same timestamp).

For every file for a message, the following Subversion properties will be set. For messages received by the svnwiki-mail program, the values will be populated based on the corresponding headers in the original message.

svnwiki:mail:headers:from
The name of the author of the message. If the message

was received through the web interface, it will be generated trying to create a valid value for a From email header.

svnwiki:mail:headers:subject
The subject of the message, if available.
svnwiki:mail:headers:date
The date in which the message was posted. No attempt

is made to decode the date, it is populated by the mail gateway with the value of the Date header or generated by the CGI application.

svnwiki:mail:headers:received
Multiple lines, each with one of the

Received headers from the message. For messages received through the web interface, it will have a single line with information about the source for the message.

svnwiki:mail:headers:x (optional)
Used only for messages received

by the svnwiki-mail gateway. Multiple lines, corresponding to headers present in the original message, that will be included verbatim as the message is sent out.

The contents of the file will be the body of the message in wiki format.

For backwards compatibility, a program will be provided that will convert all discussion files from the old format to the new in a given repository. When installing the mail extensions, admins should make sure to convert all existing discussions to the new format.

Although we are aware that the current functionality for presenting discussions (generating a page with all comments sorted from old to new) won't be very usable once the volume of messages grows, we have decided not to modify it within the scope of this specification: as long as a user doesn't subscribe to pages nor look directly at the Subversion repository, from her perspective discussions will continue to be handled as they always have. It is expected, however, that a new specification radically redesigning the discussion HTML pages will be provided after this extension has been implemented. Such new specification may also extend the format of the comments in new ways (for example, allow attachments).

SQLite tables

The mail extension will use its own SQLite table to store data about its operation. The reason to use a separate table is that the semantics of the standard SQLite table is that it is just a cache of the information contained in the wiki and thus it is OK for the administrator to remove it when generating the wiki. Instead, the email database should not be removed because it will be used by the mail extension to store information not available anywhere else.

This section describes the tables that will be used.

Email addresses

The addresses table holds the list of email address, their associated password (which is randomly generated) and a boolean indicating that the address has been confirmed.

varchar address
An email address.
bool confirmed
Has the subscription request been confirmed?
varchar password
A secret password used to authenticate the email address.
int timestamp
The time when the address was added to the table.

Subscriptions

The subscriptions tables holds the list of subscriptions. The following are its attributes:

varchar address
The email address subscribed.
varchar page
The wiki page to which the email address is subscribed.
int timestamp
The time when the subscription was created.
varchar http_address
The IP address that initiated the HTTP request that created the subscription.

One invariant that is maintained for this table is that the address exists in the addresses table and that confirmed is set to true.

Mailing lists

Whenever Svnwiki sends a message to the subscribers of a page, it will add the page to a pages table unless it already existed there. The table will have the following attributes:

varchar page
The path to the page inside the wiki (eg. svnwiki/mail-spec).
varchar id
A unique ID for the page.

The ID will be generated as follows:

Global variables

A configuration table with the following fields will be kept in memory. At any given time the table will have at most one entry.

last_revision
...
version
The string "1.0".

Managing subscriptions

This section describes how wiki users subscribe, confirm subscription, unsubscribe and confirm unsubscriptions to wiki pages.

Subscribing to a wiki page

There will be two main mechanisms by which users subscribe to wiki pages:

Submitting the form will display the following message, assuming that at least one address is actually subscribed:

5 email addresses have been subscribed to /svnwiki/mail-spec.

While we could show more information (ie. show that only 3 addresses were subscribed and that 2 were already subscribed), we have opted to provide less information for privacy reasons.

The code should be smart enough to ignore duplicates as well as confirmed addresses that were already subscribed to the page (ie. users who were already effectively subscribed to the page won't get notified of the redundant subscription).

We are fully aware that a naive design could be easily exploited by spammers: they could create a wiki page about their product and then send messages to thousands of email addresses about it. We decided to set a limit to the number of different email address that a user can subscribe to a given page in a given period of time. As long as the limit is not too low it should very rarely affect legitimate users while restricting the extent to which Svnwiki installations can be used to send spam, making them a very bad medium for spammers. To do this, whenever a subscription is requested, we verify the subscription table to make sure that in the last 24 hours the source IP address for the HTTP connection that requested the subscription has not already subscribed more address than the value specified in the svnwiki:mail:subscribe-limit (note that multiple subscriptions of the same address to multiple pages count as one). If it has, we show the following message (in addition to the above message):

The following addresses could not be subscribed because you have already reached the maximum number of subscriptions permited per day:

In the case of an edition to a page, the text describing the results will be shown in addition to the text currently shown when a page is edited.

An entry will be made in the subscriptions table for every address that wasn't already subscribed to the page. Addresses that weren't in the addresses table will be added, along with a random and unique password of 16 alphanumeric characters.

Confirmed addresses

The following message will be sent to all the addresses provided that have been confirmed:

From: list-owner@freaks-unidos.net
To: azul@freaks-unidos.net
Subject: /svnwiki/mail-spec: Subscribed

You have been subscribed by azul to the following page:

  http://wiki.freaks-unidos.net/svnwiki/mail-spec

To send messages to this page, write to:

  wiki+ID@freaks-unidos.net

To undo this request and unsubscribe from this page, go to

  https://freaks-unidos.net/cgi-bin/svnwiki/default/svnwiki/mail-spec?action=extension&extension=mail&request=unsubscribe&mail=azul@freaks-unidos.net

and enter your administration password:

  Password: PASSWORD
We recommend that you save this message for future reference, as it
may speed up the unsubscription process should you decide to do it.

If the user initiating the request wasn't authenticated, the login above (azul) will be replaced with IP 64.39.49.249 (Anonymous).

The To send messages to this page text will only be shown when the svnwiki-mail gateway is available.

We considered embedding the PASSWORD within the URL —this would certainly simplify the user interface—, but we thought it was best to make all passwords embedded in emails explicit: otherwise Alice may be more likely to forward these messages to Bob, accidentaly giving him control over her account.

Confirmation required

The following message will be sent to all the addresses provided that haven't been confirmed:

From: list-owner@freaks-unidos.net
To: azul@freaks-unidos.net
Subject: svnwiki/mail-spec: Confirmation required

We have received a subscription request from azul for this email
address for:

  http://wiki.freaks-unidos.net/svnwiki/mail-spec

In order to confirm this request and receive notifications about
modifications to this page and new messages posted to its discussion,
go to

  https://freaks-unidos.net/cgi-bin/svnwiki/default/svnwiki/mail-spec?action=extension&extension=mail&request=confirm&mail=azul@freaks-unidos.net

and enter your administration password:

  Password: PASSWORD

By doing this you will allow others subscribe you to other pages in this
site.  These new subscriptions won't require confirmation.

If you don't want to be subscribed, disregard this message: you won't be
subscribed to any page in our site.

Confirmation

When a user loads the confirm URL he gets the following message, followed by a form with a field for the password.

Confirm subscription

To confirm that you want to receive messages from pages in this wiki that you or other users subscribe you to, enter the administration password, included in the email you received:

Password: [ ... ] [Confirm]

Note that this will allow other users of this wiki to subscribe your address to any pages in this wiki that they think you may be interested in.

Submitting this form will result in any of the following actions.

Invalid password

If the password is invalid or not found, the following text will be shown:

Subscription failed

Your subscription request has failed. This may mean any of the following:

  • A long time has passed after the request was initiated.
  • The password you entered is invalid.

A new message has been sent to your email address with instructions on how to subscribe to this page, which should be completed within 14 hours.

As this implies:

Successful subscription

If the user visits the URL with the right password before his address is removed, he will see a message such as the following:

Subscription successful

Your email address azul@freaks-unidos.net has been subscribed to the following pages:

We have sent you a confirmation email message for each of these pages.

Furthermore, any users will now be able to subscribe your address to any pages in this wiki that they think you could be interested in.

Note that all the pages to which the address is subscribed will be listed.

If the address had not already been confirmed, a confirmation message will be sent to the user for each mailing list to which he is subscribed, just like the one described in section #New subscription without confirmation.

Unsubscribing

When a user clicks on one of the unsubscribe links included in all emails that Svnwiki sends to subscribers, they will be shown the following text:

Unsubscribe

Please enter the administration password for azul@freaks-unidos.net to confirm that you want to unsubscribe it from /svnwiki/mail-spec:

Password: [ ... ] [ Confirm ]

The administration password was sent in the message that confirmed your subscription to this page. If you forgot it, just click here and we will resend it to you.

If, however, the user was not subscribed to the page, the following message will be shown:

Unsubscribe

Address azul@freaks-unidos.net is currently not subscribed to /svnwiki/mail-spec.

If the user clicks in the forgot password link, he'll get a message such as:

From: list-owner@freaks-unidos.net
To: azul@freaks-unidos.net
Subject: Administration password

We received a request from azul (64.34.64.36) asking for your
administration password to be resent to you.  Here it is:

  Password: PASSWORD

If the user enters the wrong password, the following error message will be shown:

Unsubscribe: Invalid password

The administration password entered is invalid. We have sent you an email with your correct administration password. Once you receive it, please enter it here to confirm that you want to unsubscribe from /svnwiki/mail-spec:

Password: [ ... ] [ Confirm ]

If the user enters the right password, the following message will be shown:

Unsubscribed.

You have been unsubscribed from page /svnwiki/mail-spec. <Undo this operation and resubscribe>.

You are still subscribed to 4 other pages in this wiki and other users can subscribe you to more. If you want to unsubscribe from all pages in this wiki and prevent other users from automatically adding you in the future to any pages that they think you may be interested on, use the following button. Be careful: there is no simple way to Undo this operation! You'll have to confirm your subscriptions again and resubscribe to all pages.

[Unsubscribe from all pages (button)]

If the user is not subscribed to any other pages, everything after (including) the second paragraph will be replaced with:

If you want to prevent other users from automatically adding you in the future to any pages that they think you may be interested on, use the following button:

[Remove my address from this website (button)]

A message such as the following will be sent to the user, with a subject such as Unsubscribed: svnwiki/mail-spec, for every list from which he unsubscribes:

From: list-owner@freaks-unidos.net
To: azul@freaks-unidos.net
Subject: /svnwiki/mail-spec: Subscribed

You have been unsubscribed from the following page:

  http://wiki.freaks-unidos.net/svnwiki/mail-spec

If you want to subscribe in the future, you can do so by visiting
the following page:

  https://freaks-unidos.net/cgi-bin/svnwiki/default/svnwiki/mail-spec?action=extension&extension=mail&request=subscribe&mail=azul@freaks-unidos.net

Unsubscribing from all pages

If the user submits the Unsubscribe from all pages form (with the right password encoded in a hidden form field), all information about her email address (including its password) will be erased. The following message will be shown:

Unsubscribed from all pages

You have been unsubscribed from all pages in this wiki and your address has be removed from this website.

Users of this wiki won't be able to add you to any pages in the future without your confirmation.

Notifications

The notifications generated for regular pages differs from those for pages corresponding to posts in discussions in one significant characteristic. For regular pages, the notifications are designed to show the changes in the information; for posts to discussions they are oriented to show the contents. This has the following consequences:

  1. If the svnwiki-post-commit-hook program detects several new revisions for a regular file, it will send one message for each. For a given post to a discussion, however, it will only send one message, with the contents in the latest revision.
  2. Related to the above, once a notification has been sent for a file for a given post in a discussion, subsequent commits to that file will be ignored. By doing this we discourage Svnwiki users from modifying discussion posts once they have been commited (although Subversion obviously allows them to).
  3. Notifications corresponding to regular pages will include the Subversion changes log and the diff for their associated revision. On the other hand, notifications for posts to discussions will include the entire content in the post.

The last-revision field of the configuration SQL table will be used to keep track of which is the last revision in the repository for which all notifications have been sent. This allows the administrator to erase Svnwiki's last-revision.txt file —to regenerate the whole set of static pages— without having Svnwiki resend any notifications. The last-revision field will be updated by a function in the mail extension executed by the svnwiki-post-commit-hook program after all the notifications have been sent.

Modifications to pages

When a page is modified, Svnwiki will obtain a list of all revisions between the last-revision in which the page was modified. For each such revision r, for all subscribers to the file or one of its parent directories (eg. /, /svnwiki, /svnwiki/mail-spec), a message such as the following will be sent:

From: wiki+xsvnwiki-mail/notifications/ID@freaks-unidos.net
To: azul@freaks-unidos.net
Subject: r19885: Fixing some typo
List-Id: ID.notifications.wiki.freaks-unidos.net
List-Post: NO

The following page has been modified:

  http://wiki.freaks-unidos.net/svnwiki/mail-spec
Description of changes:
> Changes applied for Rodrigo (83.43.131.110) through svnwiki:
> 
> Fixing some typo

----
To cancel your subscription and stop receiving these messages go to:
https://freaks-unidos.net/cgi-bin/svnwiki/default/svnwiki/mail-spec?action=extension&extension=mail&request=unsubscribe&mail=azul@freaks-unidos.net

A diff for the commit will be included in an attachment.

Occurrences of ID in the message above will be replaced with the ID of the file or directory with the subscription that caused the message to be sent (eg. /svnwiki), not the ID of the actual page modified (eg. /svnwiki/mail-spec). As a consequence, a user subscribed to a directory will be able to create a List-Id filter to match all messages sent as a result of his subscription, but he will not be able to create a List-Id filter to match an individual page. If the user wants to create a List-Id filter for an individual page, she'll have to subscribe to that specific page.

Note that if a user is subscribed to a page and N of its parent directories, she will receive N redundant messages. We decided not to optimized the logic to only send her one, because it makes it easier for users to create appropriate filtering rules and will probably provide a better match for their expectations.

Discussion posts

When a discussion post is created or modified, Svnwiki will verify that it had not previously sent any notifications about it by comparing the revision in which it was created (not the revision in which it was last modified!) with the last-revision. If it was created at or before last-revision, it will ignore it (see numeral 2 in the above description of the differences in the way Svnwiki handles regular pages and posts to formus for the rationale). Otherwise, a message such as the following will be sent to all subscribers to the page or any of its parents directories (eg. /, /svnwiki, /svnwiki/mail-spec):

From: jorge <jorgepqmr2391hjza@gmail.com>
To: <wiki+happy-dalek@bogowiki.org>
Subject: Can we bring some sanity back???
List-Id: happy-dalek.bogowiki.org
List-Post: <wiki+happy-dalek@bogowiki.org>

I want it to be known that I believe that the new H. R. Giger-inspired
decoration is the third worst thing that could have happened to our cozy bar.
Opening my eyes the other night after finishing my daily allowance of Spice
Melange to find myself suddenly staring directly at a copy of Work 219:
Landscape XX in the roof almost made me take them eyes out, dip them in wasabi
and smash them hard against all that obsenity but, unfortunately, I was too
drunk and only managed to scratch my nose on the left a little bit and I ended
up staring at that gross piece of crap for a few more seconds.

The second worst thing, of course, is the new chairs.  They are very
uncomfortable and I've had THREE PANTS ruined as the result of dangling nails,
one time actually hurting my leg and causing me to go to the hospital as we
COULDN'T STOP THE BLEEDING in the two hours me and my friends tried.  This is
dangerous, people!  I'm still waiting for the results from my health tests, I
hope I didn't catch any nasty diseases, though I'm confident that after
pouring Pan Galactic Gargle Blaster on it I should be fine.  But still!

The first worst thing is, as everybody knows, the fact that Nelson Castillo
was allowed into the bar on 2008-02-12.  Ugh.

Can we please bring some sanity back to our pub???  I don't think translating
the menus to Klingon will make any difference.  Why Klingon and not Quenya?  I
also couldn't care less about the price of the Pan Galactic Gargle Blaster, I
don't like that strong stuff.  However, I must confess that the waitress looks
HOT with the new hairstyle.  But please, fix the obvious problems FIRST!

Jorge.

----
To cancel your subscription and stop receiving these messages go to:
https://freaks-unidos.net/cgi-bin/svnwiki/bogowiki/happy-dalek?action=extension&extension=mail&request=unsubscribe&mail=azul@freaks-unidos.net

As in the case of notifications for modifications to normal pages, the ID in the message (in the List-Id) will correspond to the ID of the page for the discussion to which the comment was posted (not the ID of the directory or file that caused the message to be delivered).

The contents sent will correspond to the latest available revision for the comment.

Mail to discussion gateway

When a message is sent to wiki+ID@freaks-unidos.net, where ID is the ID of a page, the svnwiki-mail program will verify that the source address is subscribed to the page or one of its containing directories (ie. /, /svnwiki, /svnwiki/mail-spec). If it isn't, it will simply send the following message back to the source:

Subject: V1agra cheapz!!!!!!!
From: list-owner@freaks-unidos.net
To: javiermorenodecaro@hatmiol.com
Your message has NOT been delivered to page:

  http://bogowiki.org/happy-dalek

We received a message from this address with the following headers:
> Subject: V1agra cheapz!!!!!!!
> Date: Thu Dec 13 06:37:47 CET 2007
> Message-ID: youreameanbastardhatmiol.com
It was not sent to the page because the following email address is
not subscribed:

  javiermorenodecaro@hatmiol.com

To have your message sent to all subscribers, please resend it
from an address that is subscribed to the page.

We apologize for the inconvenience, resulting from our anti-spam
measures.

The following are the received headers from the original message,
in case it helps:

Received: ...
Received: ...

Otherwise —if the address is subscribed—, it will post the message as a comment in the discussion page associated with the file.

This works as follows:

TODO: Fix this.

Tasks

Done

Pending

2748545 JP 2143076 JP Of 6233617/06 ext 104 310 884 9795

Last update: 2008-06-25 (Rev 14297)

svnwiki $Rev: 14721 $