This documentation is for Dovecot v1.x, see wiki2 for v2.x documentation.
Differences between revisions 32 and 33
Revision 32 as of 2009-07-13 19:19:21
Size: 12065
Editor: p4FEFB318
Comment:
Revision 33 as of 2009-07-15 20:43:09
Size: 12172
Editor: TimoSirainen
Comment:
Deletions are marked like this. Additions are marked like this.
Line 17: Line 17:
 * "%" works by matching any number of characters, but it stops at the hierarchy separator. Currently the separator is hardcoded to "/".  * "%" works by matching any number of characters, but it stops at the hierarchy separator. Currently the separator is hardcoded to "/", so it doesn't work correctly if you've configured separator to something else (e.g. "." is the default for Maildir).
Line 22: Line 22:
dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool.sh dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool
Line 27: Line 27:

=
=== Dovecot v1.2.x ====
Since v1.2+ expire-tool loads plugins defined in protocol imap section of dovecot.conf. However some plugins (confirmed ''imap_quota'' and ''mail_log'') fail to load. To make expire-tool working again the failing plugins have to be removed from MAIL_PLUGINS environment variable. To achieve this create ''/usr/libexec/dovecot/expire-tool.sh'' containing
=== Dovecot v1.2 ===
Since v1.2 expire-tool loads plugins defined in protocol imap section of {{{dovecot.conf}}}. However some plugins (at least ''imap_quota'' and ''mail_log'') fail to load. To make expire-tool work again the failing plugins have to be removed from MAIL_PLUGINS environment variable. To achieve this create {{{/usr/libexec/dovecot/expire-tool.sh}}}:
Line 34: Line 34:
Line 36: Line 37:
Execute Execute:
Line 40: Line 41:
Now you can use this for your cronjob Now you can use this for your cronjob:

Expire plugin

(v1.1+ only)

The expire plugin was created to keep track of mails in specific mailboxes, and expunge them when they've been there for a specified amount of time. It keeps an internal database (Berkeley DB or SQL) of all such mailboxes, so it doesn't have to go through all the mailboxes for all the users.

The expire days are counted from when the message was saved or copied to the mailbox (NOT the time message was originally received) while expire plugin was loaded. If there are existing messages in the mailbox, they'll get expunged eventually when the first message saved/copied after expire plugin was enabled gets expunged.

The save/copy date may not be exact if it's not cached in dovecot.index.cache:

  • mbox: The current lookup time is used and added to cache.
  • maildir: File's ctime is used.
  • dbox: Save/copy time is taken from the dbox file if it exists (it normally should), fallbacking to file's ctime if not.

Mailbox patterns can contain IMAP LIST command-compatible wildcards:

  • "*" works in a standard way: It matches any number of characters.
  • "%" works by matching any number of characters, but it stops at the hierarchy separator. Currently the separator is hardcoded to "/", so it doesn't work correctly if you've configured separator to something else (e.g. "." is the default for Maildir).

The expire plugin itself doesn't do anything except keep track of messages in the mailboxes. To actually perform the expunging you need to run expire-tool in for example a nightly cronjob (can be run as often as you want though):

dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool

You can also run it manually with --test parameter to see what would happen without actually changing anything. This can also be useful for debugging why it doesn't appear to be working.

Dovecot v1.2

Since v1.2 expire-tool loads plugins defined in protocol imap section of dovecot.conf. However some plugins (at least imap_quota and mail_log) fail to load. To make expire-tool work again the failing plugins have to be removed from MAIL_PLUGINS environment variable. To achieve this create /usr/libexec/dovecot/expire-tool.sh:

#!/bin/bash
MAIL_PLUGINS=${MAIL_PLUGINS//imap_quota/}
MAIL_PLUGINS=${MAIL_PLUGINS//mail_log/} 

exec ${0%.sh} "$@"

Execute:

chmod 755 /usr/libexec/dovecot/expire-tool.sh

Now you can use this for your cronjob:

dovecot --exec-mail ext /usr/libexec/dovecot/expire-tool.sh

Mail location setting problem (v1.1-v1.2)

Because of the way expire-tool is executed, you can't use user-specific %variables in mail_location. These are expanded by "dovecot" binary before it even calls expire-tool. So for example if you have:

mail_location = maildir:/var/mail/%u

expire-tool will see it as:

mail_location = maildir:/var/mail/root

This is fixed in Dovecot v2.0, but with older versions the path has to come from userdb lookup. You can either return "mail" userdb extra field or return "home" and use ~/ to point to the user's mailbox path in mail_location.

Authentication socket

expire-tool requires an auth-master socket to find users' mailboxes. See LDA for how to configure one. If you're not using the default path for the socket, you can change it by adding auth_socket_path to plugin section.

Example configuration

MySQL Backend

dovecot.conf:

protocol imap {
  mail_plugins = expire
}
protocol pop3 {
  mail_plugins = expire
}
protocol lda {
  # probably not necessary - just enables tracking messages on mailboxes
  # where messages are never saved via IMAP and never expunged
  mail_plugins = expire
}
dict {
  # NOTE: dict process currently runs as root, so this file will be owned as root.
  expire = mysql:/etc/dovecot-dict-expire.conf
}
plugin {
  # Trash and its children 7d, Spam 30d
  expire = Trash 7 Trash/* 7 Spam 30
  expire_dict = proxy::expire

  # If you have a non-default path to auth-master, set also:
  #auth_socket_path = /var/run/dovecot/auth-master
}

Create the table like this:

# for v1.1 only:
CREATE TABLE expires (
  mailbox varchar(255) not null,
  expire_stamp integer not null,
  primary key (mailbox)
);

# for v1.2+:
CREATE TABLE expires (
  username varchar(100) not null,
  mailbox varchar(255) not null,
  expire_stamp integer not null,
  primary key (username, mailbox)
);

dovecot-dict-expire.conf:

connect = host=localhost dbname=mails user=sqluser password=sqlpass

# v1.1 only:
table = expires
select_field = expire_stamp
where_field = mailbox
username_field = not_used

# v1.2+ only:
map {
  pattern = shared/expire/$user/$mailbox
  table = expires
  value_field = expire_stamp

  fields {
    username = $user
    mailbox = $mailbox
  }
}

PostgreSQL Backend

Like MySQL configuration above, but you'll also need to create a trigger:

-- v1.1+:
CREATE OR REPLACE FUNCTION merge_expires() RETURNS TRIGGER AS $$
BEGIN
  IF exists(SELECT 1 FROM expires WHERE mailbox = NEW.mailbox) THEN
    UPDATE expires SET expire_stamp = NEW.expire_stamp
      WHERE mailbox = NEW.mailbox;
    RETURN NULL;
  ELSE
    RETURN NEW;
  END IF;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER mergeexpires BEFORE INSERT ON expires
   FOR EACH ROW EXECUTE PROCEDURE merge_expires();

BDB Backend

The Berkeley DB code in Dovecot doesn't work very well, so this configuration isn't recommended. Use SQL instead.

protocol imap {
  mail_plugins = expire
}
protocol pop3 {
  mail_plugins = expire
}
protocol lda {
  # probably not necessary - just enables tracking messages on mailboxes
  # where messages are never saved via IMAP and never expunged
  mail_plugins = expire
}
dict {
  # NOTE: dict process currently runs as root, so this file will be owned as root.
  expire = db:/var/lib/dovecot/expire.db
}
plugin {
  # Trash and its children 7d, Spam 30d
  expire = Trash 7 Trash/* 7 Spam 30
  expire_dict = proxy::expire

  # If you have a non-default path to auth-master, set also:
  #auth_socket_path = /var/run/dovecot/auth-master
}

Alternative dbox directory expiration

Expire plugin can also be used to move old mail files to dbox alternative directory. The idea behind this is that old mails are accessed rarely, so the alternative directory may be located on a cheaper storage with lower I/O capabilities. expire_altmove setting can be used to configure this:

mail_location = dbox:~/dbox:ALT=/altstorage/%d/%n/dbox
# mail_plugin and dict settings as in above example
plugin {
  expire = Trash 7 Trash/* 7 Spam 30
  # Move all mails to slow storage after 31 days (so Trash/Spam is never moved)
  expire_altmove = * 31
  expire_dict = proxy::expire
}

Example #1 timeline

Let's say Trash is configured to expire in 7 days and today is 2009-07-10. Initially the database and the Trash mailbox is empty.

User moves the first message to Trash. The expires table is updated:

mysql> select mailbox, from_unixtime(expire_stamp), username from expires;
+---------+-----------------------------+----------+
| mailbox | from_unixtime(expire_stamp) | username |
+---------+-----------------------------+----------+
| Trash   | 2009-07-17 15:57:36         | tss      | 
+---------+-----------------------------+----------+

The expire_stamp contains the date when expire-tool will look into that mailbox and try to find messages to expunge. Until then it skips the mailbox.

A day later user moves another message to Trash. The expire_stamp isn't updated, because the second message's save date is newer than the first one's. Checking Trash's contents via IMAP you can see something like (X-SAVEDATE requires v1.2.2+):

1 fetch 1:* (internaldate x-savedate)
* 1 FETCH (INTERNALDATE "16-Dec-2008 09:52:38 -0500" X-SAVEDATE "10-Jul-2009 15:57:36 -0400")
* 2 FETCH (INTERNALDATE "29-Jun-2003 23:20:09 -0400" X-SAVEDATE "11-Jul-2009 16:03:11 -0400")
1 OK Fetch completed.

Note how the message's INTERNALDATE (received date) can be very old compared to the save date. Now, running expire-tool --test:

Info: tss/Trash: stop, expire time in future: Fri Jul 17 15:57:36 2009

So it does nothing, because the expire time is in future. Fast forward 6 more days into future. Running expire-tool --test:

Info: tss/Trash: seq=1 uid=1: Expunge
Info: tss/Trash: timestamp 1247860656 (Fri Jul 17 15:57:36 2009) -> 1247947391 (Sat Jul 18 16:03:11 2009)

The first message would be expunged and the second message's timestamp would become the new expire_stamp in database. After running expire-tool without --test, the database is updated:

mysql> select mailbox, from_unixtime(expire_stamp), username from expires;
+---------+-----------------------------+----------+
| mailbox | from_unixtime(expire_stamp) | username |
+---------+-----------------------------+----------+
| Trash   | 2009-07-18 16:03:11         | tss      | 
+---------+-----------------------------+----------+

Also you can see the first message has been expunged from Trash:

2 fetch 1:* (internaldate x-savedate)
* 1 FETCH (INTERNALDATE "29-Jun-2003 23:20:09 -0400" X-SAVEDATE "11-Jul-2009 16:03:11 -0400")
2 OK Fetch completed.

Example #2 timeline

Again you have Trash configured for 7 days, but this time you have an existing message there before expire plugin has been enabled. Initially the expire database is empty. Today is 2009-07-20.

1 fetch 1:* (internaldate x-savedate)
* 1 FETCH (INTERNALDATE "29-Jun-2003 23:20:09 -0400" X-SAVEDATE "11-Jul-2009 16:03:11 -0400")
1 OK Fetch completed.

If you run expire-tool, you'll notice that it does nothing for the mailbox. There's nothing in expire database, so expire-tool doesn't even mention it when running with --test.

After user moves the first message to Trash, the database gets updated:

mysql> select mailbox, from_unixtime(expire_stamp), username from expires;
+---------+-----------------------------+----------+
| mailbox | from_unixtime(expire_stamp) | username |
+---------+-----------------------------+----------+
| Trash   | 2009-07-27 16:32:11         | tss      | 
+---------+-----------------------------+----------+

The messages in Trash are:

2 fetch 1:* (internaldate x-savedate)
* 1 FETCH (INTERNALDATE "29-Jun-2003 23:20:09 -0400" X-SAVEDATE "11-Jul-2009 16:03:11 -0400")
* 2 FETCH (INTERNALDATE "16-Dec-2002 11:02:39 -0500" X-SAVEDATE "20-Jul-2009 16:32:11 -0400")
2 OK Fetch completed.

So the first message should be expiring already, right? No. It doesn't because the timestamp in database is still in future. expire-tool --test says:

Info: tss/Trash: stop, expire time in future: Mon Jul 27 16:32:11 2009

OK, let's see what happens when we finally reach July 27th:

Info: tss/Trash: seq=1 uid=3: Expunge
Info: tss/Trash: seq=2 uid=4: Expunge
Info: tss/Trash: no messages left

They both got expunged! The expire database's timestamp simply tells expire-tool when to start looking into messages in that mailbox. After that expire-tool looks at the actual save dates and figures out which messages exactly need to be expunged.

After running expire-tool without --test you'll see that the Trash mailbox is empty and the database row is deleted.

v1.0 cronjob equivalent

For Dovecot v1.0, this can be accomplished by running a daily shell script:

# delete 30 day old mails
find /var/virtualmail/ -regex '.*/\.\(Trash\|Junk\)\(/.*\)?\/\(cur\|new\)/.*' -type f  -ctime +30  -delete
# or -exec rm '{}' \; instead of -delete

Using ctime means that messages are deleted 30 days after they've been moved to Trash. If mtime was used instead, it would mean the message is deleted 30 days after its original creation, which could be immediately.

None: Plugins/Expire (last edited 2012-02-06 12:38:22 by TimoSirainen)