This documentation is for Dovecot v1.x, see wiki2 for v2.x documentation.
Differences between revisions 39 and 40
Revision 39 as of 2006-05-18 09:40:39
Size: 12575
Comment: Turn folder internal data into a level 3 heading
Revision 40 as of 2006-05-18 11:39:54
Size: 15663
Comment: general overhaul
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
Dovecot is one of the easiest IMAP servers to migrate to because of its very powerful configuration options. Dovecot can store email in both MBOX and MAILDIR formats making it compatible with most other mail storage formats. It is very flexible as to where it stores the email. Dovecot supports many methods for storing user data and password authentication. You can set it up to authenticate against multiple databases. Dovecot is one of the easiest IMAP servers to migrate to because of its very powerful configuration options. Dovecot can store email in both MBOX and MAILDIR formats, making it compatible with most other mail storage formats. Dovecot is also very flexible as to where it stores the email. It supports many methods for storing user data and password authentication. You can set it up to authenticate against multiple databases.
Line 13: Line 13:
When migrating from one IMAP server to another, you should make sure that these are preserved:

 * Mailbox subscription list
  * User would be able to manually subscribe them again if you don't want to mess with it.
 * Message UIDs
  * If UIDs are lost, at the minimum clients' message cache gets cleaned
When migrating from one IMAP/POP3 server to another, you should make sure that these are preserved:

 1. Message flags
  * Lost flags can be really annoying, you most likely want to avoid it.
 1. Message UIDs (and for IMAP, the UIDVALIDITY value)
  * If UIDs are lost, at the minimum clients' message cache gets cleaned and messages are re-downloaded as new.
Line 20: Line 20:
 * Message flags
  * Lost flags can be really annoying, you most likely want to avoid it.

Here's the more server-specific instructions:
 1. Mailbox subscription list (only for IMAP, POP3 does not have a notion of folders)
  * Users would be able to manually subscribe them again if you don't want to mess with it.

Here are the more server-specific instructions:
Line 27: Line 27:
By default UW-IMAP allows access to whole home directory, and many people have chosen to store their mails in mail/ directory. This usually means that IMAP clients have set "IMAP namespace" to "mail/", which doesn't work well with Dovecot, as Dovecot by default uses mail/ directory directly. So if IMAP namespace is kept as "mail/", Dovecot would try to access "~/mail/mail/" directory.

So, removing the prefix from IMAP clients would be the first step. Next problem is that subscribed mailboxes are listed as "mail/box" or "~/mail/box" or "~user/mail/box" in subscriptions file. You'd have to remove the mail/ directory part from all of these. The subscriptions file name is also different, UW-IMAP uses `.mailboxlist` while Dovecot uses `.subscriptions`.
By default UW-IMAP allows access to whole home directory, and many people have chosen to store their mails in the `mail/` directory. This usually means that IMAP clients have set "IMAP namespace" to `"mail/"`, which doesn't work well with Dovecot, as Dovecot uses the `mail/` directory directly by default. So if the IMAP namespace were kept as `"mail/"`, Dovecot would try to access the `~/mail/mail/` directory.

Thus, removing the prefix from IMAP clients would be the first step. Next problem is that subscribed mailboxes are listed as `"mail/box"` or `"~/mail/box" or "~user/mail/box"` in subscriptions file. You'd have to remove the `mail/` directory part from all of these. The subscriptions file name is also different, UW-IMAP uses `.mailboxlist` while Dovecot uses `.subscriptions`.
Line 38: Line 38:
mbox_locks = dotlock fcntl
}}}

If you want to make a transparent migration to Dovecot without having to change the configuration on hundreds of client systems, you will need a slightly different configuration.

 * If your clients have the server prefix set to something like "~/mail", this will not work unless you enable mail_full_filesystem_access in your Dovecot configuration. Dovecot will otherwise reject mailbox names or prefixes that start with "~". Alternative to this with 1.0-tests is to [wiki:Namespaces set up multiple hidden namespaces].

 * You can either rename your ".mailboxlist" file to ".subscriptions" for all you users using [wiki:uw2dovecot.sh uw2dovecot.sh], or change the definition of SUBSCRIPTION_FILE_NAME in ''src/lib-storage/subscription-file/subscription-file.c''. In 1.0-test, this value is defined in ''src/lib-storage/subscription-fileindex/maildir/maildir-storage.h'' and ''src/lib-storage/subscription-fileindex/mbox/maildir-storage.h''. .subscriptions is located inside of the mail directory, whereas it is possible .mailboxlist was outside of it (in user's home directory).

 * If ~/mbox file exists, UW-IMAP moves mails from /var/mail/user there. Currently Dovecot doesn't support this feature, so you'll have to either move everyone's mails to ~/mbox and reconfigure MTA/LDA to store mails there, or alternatively get the ~/mbox users to move their mails back to /var/mail/user. This feature may be implemented later, but it's not planned in near future.

 * You should read [http://people.redhat.com/rkeech/maildir-migration.txt]. This document describes the migration process using scripts if you want to migrate from UW_IMAP with mbox to Dovecot using maildir (which will increase performance). Also see the "Converting Mailbox Formats" section below.

=== DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA ===

Only concerns Dovecot 0.99.x versions, 1.0-releases use them:


Every folder on a system using UW-IMAP has a system message in the top with the subject "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA". This system message has information about the folder and its state. Dovecot 0.99.x does not use these system messages to anything so you can delete them without any problems - unless you have users using Pine together with an IMAP or POP3 mailclient.

= UW-POP3 =

By default Dovecot generates POP3 UIDs differently than
UW-IMAP. If you want to preserve them, with 0.99.x you'll have to
[http://dovecot.org/patches/
pop3-uidl-uwimap.patch patch Dovecot]. With 1.0-tests you can use:

{{{
pop
3_uidl_format = %08Xv%08Xu
}}}

Note: Confirm that old and new
UIDLs match. 0.99.x may not have the same uidl as UW-IMAP. Version 1.0 appears to be the same as UW-IMAP. Check with:

{{{
mbox_locks = dotlock fcntl}}}

If you want to make a '''transparent migration''' to Dovecot without having to change the configuration on hundreds of client systems, you will need a slightly different configuration:

 * If your clients have the server prefix set to something like `"~/mail"`, you may want to enable `mail_full_filesystem_access` in your Dovecot configuration. Dovecot will otherwise reject mailbox names or prefixes that start with "~". An alternative to this with 1.0-tests is to [wiki:Namespaces set up multiple hidden namespaces].

 * You can either rename your `.mailboxlist` file to `.subscriptions` for all your users using [wiki:uw2dovecot.sh uw2dovecot.sh], or change the definition of SUBSCRIPTION_FILE_NAME in ''src/lib-storage/subscription-file/subscription-file.c''. In 1.0-test, this value is defined in ''src/lib-storage/subscription-fileindex/maildir/maildir-storage.h'' and ''src/lib-storage/subscription-fileindex/mbox/maildir-storage.h''. .subscriptions is located inside of the mail directory, whereas it is possible .mailboxlist was outside of it (in user's home directory).

 * If an `~/mbox` file exists, UW-IMAP moves mails from `/var/mail/user` there. Currently Dovecot doesn't support this feature, so you'll have to either move everyone's mails to `~/mbox` and reconfigure the MTA, MDA or LDA to store mails there, or alternatively get the `~/mbox` users to move their mails back to `/var/mail/user`. The mbox feature may be implemented later, but it's not planned in near future.

 * You should read [http://people.redhat.com/rkeech/maildir-migration.txt]. This document describes the migration process using scripts if you want to migrate from UW_IMAP with mbox to Dovecot using maildir (which will increase performance). Also see the [#convert Converting Mailbox Formats] section below.

== DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA ==

Every folder on a system using UW-IMAP has a system message in the top with the subject "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA". This system message has information about the folder and its state.

Dovecot 1.0 releases use these system messages. Keep them.

Dovecot 0.99.x does not use these system messages to anything so you can delete them without any problems - unless you have users using PINE together with an IMAP or POP3 mail client.

= UW-POP3 (UW-IMAP's POP3 wrapper, ipop3d) =

By default
Dovecot generates POP3 UIDs differently than UW-IMAP, which causes POP3 clients to not recognize them and redownload messages they had downloaded previously.

If you want Dovecot to use the same UID format
as UW-IMAP/ipop3d so that clients can recognize them, your required action depends on the Dovecot version.

 Dovecot 0.99.X::
 With Dovecot 0.99.x you'll have to [http://dovecot.org/patches/old/pop3-uidl-u
wimap.patch patch Dovecot].

 Dovecot 1.0 (including test and betas)
::
 With Dovecot 1.0 versions you can set this configuration option:
{{{
pop3_uidl_format = %08Xv%08Xu}}}

'''Note:''' Confirm that old and new UIDLs match after the patch or reconfiguration. Check with: {{{
Line 73: Line 76:
}}} quit}}}
Line 77: Line 80:
VIMAP or [http://vimap.sourceforge.net/ Virtual Imap] was a patched version of WU-IMAP that was modified to work with [http://www.solucorp.qc.ca/linuxconf/ Linuxconf's] virtual email configuration. The project doesn't seem to have much life and [mailto:patrik.hall@multitronic.fi Patrick Hall] who wrote the patch is thrilled that Dovecot can now be configured to be fully compatible. Details of how to do this are described in the [wiki:VirtualUsers Virtual Users] section.

= Courier =

Courier by default uses "INBOX." as private IMAP namespace, so it has exactly the same problems as described with UW-IMAP above. Again, if you want the migration to be transparent for IMAP clients see [wiki:Namespaces Namespaces] how to get namespaces to be Courier-compatible.
VIMAP or [http://vimap.sourceforge.net/ Virtual Imap] was a patched version of UW-IMAP that was modified to work with [http://www.solucorp.qc.ca/linuxconf/ Linuxconf's] virtual email configuration. The project doesn't seem to have much life and [mailto:patrik.hall@multitronic.fi Patrick Hall] who wrote the patch is thrilled that Dovecot can now be configured to be fully compatible. Details of how to do this are described in the [wiki:VirtualUsers Virtual Users] section.

= Courier, Courier-IMAP =

Courier by default uses "INBOX." as private IMAP namespace, so it has exactly the same problems as described with UW-IMAP above. Again, if you want the migration to be transparent for IMAP clients, see [wiki:Namespaces Namespaces] how to set namespaces to be Courier-compatible.
Line 85: Line 88:
 * Courier's message flags are compatible with Dovecot (as it's specified by Maildir specification)
 * Courier's message keywords implementation isn't Dovecot compatible and there's currently no easy way to migrate them.
 * Courier's message flags are compatible with Dovecot (as they are specified by the Maildir specification)
 * Courier's message keywords implementation isn't Dovecot compatible and there is currently no easy way to migrate them.
Line 90: Line 93:
Settings:

{{{
Settings: {{{
Line 96: Line 97:
default_mail_env = maildir:~/
}}}
default_mail_env = maildir:~/}}}

For POP3 UIDL compatibility with Courier, the Dovecot configuration depends on the Courier version used:

 Courier version 0::
 {{{
# Courier version 0 (using maildir filenames)
pop3_uidl_format = %f}}}

 Courier version 1::
 {{{
# Courier version 1 (UID)
pop3_uidl_format = %u}}}

 Courier version 2 and newer::
 {{{
# Courier version 2 (UIDVALIDITY and UID - you most likely want this)
pop3_uidl_format = %v-%u}}}

= Cyrus =

See [http://madness.at/projects/ cyrus2courier], it is Dovecot-compatible. Cyrus2courier is also mirrored at [http://dovecot.org/tools/ dovecot.org].
Line 101: Line 122:
{{{
# Courier version 0 (using maildir filenames)
pop3_uidl_format = %f
# Courier version 1 (UID)
pop3_uidl_format = %u
# Courier version 2 (UIDVALIDITY and UID - you most likely want this)
pop3_uidl_format = %v-%u
}}}

= Cyrus =

See [http://madness.at/projects/ cyrus2courier], it's Dovecot-compatible. Also mirrored at [http://dovecot.org/tools/ dovecot.org].

For POP3 UIDL compatibility, use one of:

{{{
 Cyrus versions up to v2.1.3:: {{{
Line 118: Line 124:
pop3_uidl_format = %u pop3_uidl_format = %u}}}
 Cyrus versions v2.1.4 and newer:: {{{
Line 120: Line 127:
pop3_uidl_format = %v.%u
}}}

See [http://www.majid.info/mylos/weblog/2006/03/08-1.html] for an alternative migration script (in Python) for Cyrus IMAPd 2.2, which will preserve delivery timestamps and read status.

Also, there exists a script called
IMAPSync [http://freshmeat.net/projects/imapsync] which does a nice job of syncing from one IMAP server to another without caring what IMAP server is runing on either end. It has been effective for various migrations for us so far, and worked well for migrating from a Cyrus IMAP server to Dovecot. Here's an example of how you might run the script for a single user:

{{{
imapsync --host1 192.168.1.57 --authmech1 LOGIN --user1 leah@example.com --password1 secret --host2 localhost --authmech2 LOGIN --user2 leah@example.com --password2 secret --syncinternaldates
}}}

This is quite easy to script for a number of users, assuming you have their passwords. Even if you do not, the script also supports logging in as an admin user that has the ability to copy message for sub users, and a variety of other authentication options.

= Other POP3 servers =
pop3_uidl_format = %v.%u}}}

See [http://www.majid.info/mylos/weblog/2006/03/08-1.html] for an alternative migration script (in Python) for Cyrus IMAPd 2.2, which will preserve delivery timestamps and "read" status.

There is also a script called [http://freshmeat.net/projects/imapsync
IMAPsync] which does a nice job of synching from one IMAP server to another without caring what IMAP server is runing on either end. It has been effective for various migrations for us so far, and worked well for migrating from a Cyrus IMAP server to Dovecot. Here's an example of how you might run the script for a single user:

{{{
imapsync --syncinternaldates \
 
--host1 192.168.1.57 --authmech1 LOGIN --user1 leah@example.com --password1 secret \
  
--host2 127.0.0.1 --authmech2 LOGIN --user2 leah@example.com --password2 secret }}}

This is quite easy to script this for a number of users, assuming you have their passwords. Even if you do not, the script also supports logging in as an admin user that has the ability to copy message for sub users, and a variety of other authentication options.

= Other POP3 servers (X-UIDL) =
Line 148: Line 155:
[[Anchor(convert)]]
Line 152: Line 160:
If you are using WU-IMAP and using the MBX format you will need to convert it to MBOX format. The conversion process isn't pretty, but here's a script that works. You will need to get and compile the mailutil program from the WU-IMAP web site.

{{{
#! /bin/bash
If you are using UW-IMAP and using the MBX format, you will need to convert it to MBOX format. The conversion process isn't pretty, but here is a script that works. You will need to get and compile the mailutil program from the UW-IMAP web site.

'''WARNING:''' Script versions listed here before 2006-05-18 11:40 UTC converted into mbx rather than mbox format, i. e. older versions did the opposite of what they were supposed to do. Use the version below:


{{{
#! /bin/sh
Line 158: Line 168:
# Usage mbx-convert <filename>
# This code assumes a user named "marc". Change to any real user on your system.
# overhauled by Matthias Andree, 2006
# Usage: mbx-convert <filename>
# This code assumes there a user named "marc" with the primary group "marc".
#
Change to any real user on your system.
Line 162: Line 174:
# abort on error
set -e

user=marc
group=marc
homedir=/home/$user

if [ $# -ne 1 ] ; then
  echo >&2 "Usage: $0 <filename>"
  exit 1
fi

# set up automatic cleanup
trap 'rm -f "${homedir}"/in.$$ "${homedir}"/out.$$' 0
Line 163: Line 190:
cp "$*" /home/marc/in
chown marc:marc /home/marc/in

# Run mailutil to convert as the user other than root 
cp "$1" "${homedir}/in.$$"
chown "$user":"$group" "${homedir}/in.$$"

# Run mailutil to convert as the user other than root
Line 168: Line 195:
su - marc -c "/usr/bin/mailutil copy in \#driver.mbx/out"

# cat instead of copy preserves the original owner and permissions
cat /home/marc/out > "$*"

# cleanup
rm /home/marc/in
rm /home/marc/out
}}}
su "$user" -c "mailutil copy in.$$ \#driver.unix/out.$$"

# create new file with same permissions/owner as old
cp -p "$1" "${1}.new"
# cat instead of copy leaves the original owner and permissions alone
if cat "${homedir}/out.$$" >"${1}.new" ; then
  # cat succeeded, rename file into place
  mv "${1}.new" "$1"
else
  # cat failed, remove temp file
  rm -f "${1}.new"
  exit 1
fi}}}
Line 188: Line 219:
'''User comments:'''
 * Is this hassle actually necessary? I have run mailutil as root like this (mailutil as provided by the PINE 4.61 package for SUSE Linux 10.0): {{{mailutil copy /tmp/foo.mbx.orig '#driver.unix//tmp/foo.test'}}} and did not encounter any problems. ''-- MatthiasAndree, 2006-05-18''
Line 192: Line 226:
Check also the '''User-Contributed Maildir Support''' section [http://www.qmail.org/top.html#usersoft here] for more choices.

Example (user's mail in ~someuser/mail and inbox in /var/mail/someuser):
Check also the '''User-Contributed Maildir Support''' section [http://www.qmail.org/top.html#usersoft on the qmail community site] for more choices.

Example (user's mail in `~someuser/mail` and inbox in `/var/mail/someuser`):
Line 199: Line 233:
mv mail mail.old
}}}

Now the mail will be in ~someuser/Maildir

If you are coming from uw-imap, you also need to do:
{{{
cp -p .mailboxlist Maildir/subscriptions
}}}
mv mail mail.old}}}

Now the mail will be in `~someuser/Maildir`.

If you are migrating from UW-IMAP, you also need to do:
{{{
cp -p .mailboxlist Maildir/subscriptions}}}
Line 212: Line 244:
mv subscriptions.new subscriptions
}}}

Don't forget to update "default_mail_env" in dovecot.conf (as needed)
mv subscriptions.new subscriptions}}}

Don't forget to update `default_mail_env` in dovecot.conf (as needed)
Line 221: Line 252:
Use the formail program that comes with [http://www.procmail.org/ procmail]. Here is a simple script showing how this works. Change to the user's home directory (the parent of Maildir) and run the script with two arguments: the mailbox name, and the output mbox filename.

{{{
#!/usr/local/bin/perl
Use the reformail program that comes with [http://www.courier-mta.org/maildrop/ maildrop]. You can also use the formail program that comes with [http://www.procmail.org/ procmail]. Here is a simple script showing how this works.

To use it, adjust the script to invoke the right command according to your system.

Then {{{cd}}} to the user's home directory (one level above `Maildir`) and run the script with two arguments: the mailbox name (You can use "." for the top-level folder), and the output mbox filename, for example: {{{
cd ~hans
perl dw-maildirtombox.pl . >/tmp/hans-inbox
perl dw-maildirtombox.pl Sent >/tmp/hans-sent}}}

'''WARNING:''' versions of the script provided here before 2006-05-18 11:40 UTC did not save the new/ directory contents, i. e. they missed unread messages!

{{{
#!/usr/bin/env perl

# dw-maildirtombox.pl
# dw = Dovecot Wiki :-)

# NOTE! The output file must not contain single quotes (')!

# figure out which program to run
$cmd="reformail -f1";
system("$cmd </dev/null >/dev/null 2>/dev/null") == 0 or $cmd="formail";
system("$cmd </dev/null >/dev/null 2>/dev/null") == 0
or die "cannot find reformail or formail on your \$PATH!\nAborting";
Line 228: Line 279:
if (($outputfile eq '') || ($dir eq '') || (!stat("Maildir/$dir")))
{ die "./archivemail.pl mailbox outputfile\n"; }

opendir(MAILDIR,"Maildir/$dir/cur");
while($name=readdir(MAILDIR))
{
 if (($name ne '.') && ($name ne '..'))
 {
  system("cat Maildir/$dir/cur/$name | formail >> $outputfile");
 }
if (($outputfile eq '') || ($dir eq ''))
{ die "Usage: ./archivemail.pl mailbox outputfile\nAborting"; }

if (!stat("Maildir/$dir/cur") || !stat("Maildir/$dir/new"))
{ die "Maildir/$dir is not a maildir.\nAborting"; }

@files = (<Maildir/$dir/cur/*>,<Maildir/$dir/new/*>);
foreach $file (@files) {
  next unless -f $file; # skip non-regular files
  next unless -s $file; # skip empty files
  next unless -r $file; # skip unreadable files
  $file =~ s/'/'"'"'/; # escape ' (single quote)
  $run = "cat '$file' | $cmd >>'$outputfile'";
  system($run) == 0 or warn "cannot run \"$run\".";
Line 239: Line 294:
closedir(MAILDIR);

TableOfContents

Migration to Dovecot

Dovecot is one of the easiest IMAP servers to migrate to because of its very powerful configuration options. Dovecot can store email in both MBOX and MAILDIR formats, making it compatible with most other mail storage formats. Dovecot is also very flexible as to where it stores the email. It supports many methods for storing user data and password authentication. You can set it up to authenticate against multiple databases.

Migration involves several separate tasks. You either need to convert your data or make Dovecot read your existing data. Dovecot is very good at being compatible and configurable so that it is likely to read your existing mail folders and user and password configurations. Tasks for conversion include:

  • Where and how to store the email
  • User configuration - home directory structure - UID/GID - password storage
  • Authentication of users

When migrating from one IMAP/POP3 server to another, you should make sure that these are preserved:

  1. Message flags
    • Lost flags can be really annoying, you most likely want to avoid it.
  2. Message UIDs (and for IMAP, the UIDVALIDITY value)
    • If UIDs are lost, at the minimum clients' message cache gets cleaned and messages are re-downloaded as new.
    • Some IMAP clients store metadata by assigning it to specific UID, if UIDs are changed these will be lost.
  3. Mailbox subscription list (only for IMAP, POP3 does not have a notion of folders)
    • Users would be able to manually subscribe them again if you don't want to mess with it.

Here are the more server-specific instructions:

UW-IMAP

By default UW-IMAP allows access to whole home directory, and many people have chosen to store their mails in the mail/ directory. This usually means that IMAP clients have set "IMAP namespace" to "mail/", which doesn't work well with Dovecot, as Dovecot uses the mail/ directory directly by default. So if the IMAP namespace were kept as "mail/", Dovecot would try to access the ~/mail/mail/ directory.

Thus, removing the prefix from IMAP clients would be the first step. Next problem is that subscribed mailboxes are listed as "mail/box" or "~/mail/box" or "~user/mail/box" in subscriptions file. You'd have to remove the mail/ directory part from all of these. The subscriptions file name is also different, UW-IMAP uses .mailboxlist while Dovecot uses .subscriptions.

Dovecot uses UW-IMAP compatible UID and message flag headers in mboxes, so that's not a problem.

Settings:

default_mail_env = mbox:~/mail:INBOX=/var/mail/%u
# make sure mbox_locks are the same with all software that accesses your mboxes
mbox_locks = dotlock fcntl

If you want to make a transparent migration to Dovecot without having to change the configuration on hundreds of client systems, you will need a slightly different configuration:

  • If your clients have the server prefix set to something like "~/mail", you may want to enable mail_full_filesystem_access in your Dovecot configuration. Dovecot will otherwise reject mailbox names or prefixes that start with "~". An alternative to this with 1.0-tests is to [wiki:Namespaces set up multiple hidden namespaces].

  • You can either rename your .mailboxlist file to .subscriptions for all your users using [wiki:uw2dovecot.sh uw2dovecot.sh], or change the definition of SUBSCRIPTION_FILE_NAME in src/lib-storage/subscription-file/subscription-file.c. In 1.0-test, this value is defined in src/lib-storage/subscription-fileindex/maildir/maildir-storage.h and src/lib-storage/subscription-fileindex/mbox/maildir-storage.h. .subscriptions is located inside of the mail directory, whereas it is possible .mailboxlist was outside of it (in user's home directory).

  • If an ~/mbox file exists, UW-IMAP moves mails from /var/mail/user there. Currently Dovecot doesn't support this feature, so you'll have to either move everyone's mails to ~/mbox and reconfigure the MTA, MDA or LDA to store mails there, or alternatively get the ~/mbox users to move their mails back to /var/mail/user. The mbox feature may be implemented later, but it's not planned in near future.

  • You should read [http://people.redhat.com/rkeech/maildir-migration.txt]. This document describes the migration process using scripts if you want to migrate from UW_IMAP with mbox to Dovecot using maildir (which will increase performance). Also see the [#convert Converting Mailbox Formats] section below.

DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA

Every folder on a system using UW-IMAP has a system message in the top with the subject "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA". This system message has information about the folder and its state.

Dovecot 1.0 releases use these system messages. Keep them.

Dovecot 0.99.x does not use these system messages to anything so you can delete them without any problems - unless you have users using PINE together with an IMAP or POP3 mail client.

UW-POP3 (UW-IMAP's POP3 wrapper, ipop3d)

By default Dovecot generates POP3 UIDs differently than UW-IMAP, which causes POP3 clients to not recognize them and redownload messages they had downloaded previously.

If you want Dovecot to use the same UID format as UW-IMAP/ipop3d so that clients can recognize them, your required action depends on the Dovecot version.

Dovecot 0.99.X

With Dovecot 0.99.x you'll have to [http://dovecot.org/patches/old/pop3-uidl-uwimap.patch patch Dovecot].

Dovecot 1.0 (including test and betas)

With Dovecot 1.0 versions you can set this configuration option:

pop3_uidl_format = %08Xv%08Xu

Note: Confirm that old and new UIDLs match after the patch or reconfiguration. Check with:

telnet localhost 110
user test
pass test
uidl
quit

VIMAP

VIMAP or [http://vimap.sourceforge.net/ Virtual Imap] was a patched version of UW-IMAP that was modified to work with [http://www.solucorp.qc.ca/linuxconf/ Linuxconf's] virtual email configuration. The project doesn't seem to have much life and [mailto:patrik.hall@multitronic.fi Patrick Hall] who wrote the patch is thrilled that Dovecot can now be configured to be fully compatible. Details of how to do this are described in the [wiki:VirtualUsers Virtual Users] section.

Courier, Courier-IMAP

Courier by default uses "INBOX." as private IMAP namespace, so it has exactly the same problems as described with UW-IMAP above. Again, if you want the migration to be transparent for IMAP clients, see [wiki:Namespaces Namespaces] how to set namespaces to be Courier-compatible.

  • Courier's courierimapsubscribed is compatible with Dovecot's subscriptions file, just rename it and remove the "INBOX." prefixes. (NOTE: with 0.99.x it's .subscriptions, with 1.0 it's subscriptions without the dot)

  • Courier's courierimapuiddb is compatible with Dovecot's dovecot-uidlist file, just rename it.

  • Courier's message flags are compatible with Dovecot (as they are specified by the Maildir specification)
  • Courier's message keywords implementation isn't Dovecot compatible and there is currently no easy way to migrate them.

See also the [http://bendiken.net/scripts/ courier2dovecot] script by Arto Bendiken.

Settings:

# normal home directories
default_mail_env = maildir:~/Maildir
# for virtual users
default_mail_env = maildir:~/

For POP3 UIDL compatibility with Courier, the Dovecot configuration depends on the Courier version used:

Courier version 0
# Courier version 0 (using maildir filenames)
pop3_uidl_format = %f
Courier version 1
# Courier version 1 (UID)
pop3_uidl_format = %u
Courier version 2 and newer
# Courier version 2 (UIDVALIDITY and UID - you most likely want this)
pop3_uidl_format = %v-%u

Cyrus

See [http://madness.at/projects/ cyrus2courier], it is Dovecot-compatible. Cyrus2courier is also mirrored at [http://dovecot.org/tools/ dovecot.org].

For POP3 UIDL compatibility, use one of:

Cyrus versions up to v2.1.3
#  Cyrus (old format - up to v2.1.3)
pop3_uidl_format = %u
Cyrus versions v2.1.4 and newer
#  Cyrus (new format - v2.1.4 and above)
pop3_uidl_format = %v.%u

See [http://www.majid.info/mylos/weblog/2006/03/08-1.html] for an alternative migration script (in Python) for Cyrus IMAPd 2.2, which will preserve delivery timestamps and "read" status.

There is also a script called [http://freshmeat.net/projects/imapsync IMAPsync] which does a nice job of synching from one IMAP server to another without caring what IMAP server is runing on either end. It has been effective for various migrations for us so far, and worked well for migrating from a Cyrus IMAP server to Dovecot. Here's an example of how you might run the script for a single user:

imapsync --syncinternaldates \
  --host1 192.168.1.57 --authmech1 LOGIN --user1 leah@example.com --password1 secret \
  --host2 127.0.0.1    --authmech2 LOGIN --user2 leah@example.com --password2 secret 

This is quite easy to script this for a number of users, assuming you have their passwords. Even if you do not, the script also supports logging in as an admin user that has the ability to copy message for sub users, and a variety of other authentication options.

Other POP3 servers (X-UIDL)

Some POP3 servers generate the X-UIDL header. To use this header when available and ease migration, use:

# Use X-UIDL header if available
pop3_reuse_xuidl = yes

Different POP3 servers generate UIDs differently. If you want to preserve them to avoid users downloading their mails twice, you'll need to figure out how the server generates the UID and patch Dovecot accordingly to do the same.

Here is a list of POP3 servers and how they generate their UIDs. Please update if you know more:

  • [http://www.openwall.com/popa3d/ popa3d] Generates MD5 sum from a couple of headers. Dovecot uses compatible MD5 sums internally but converts them into strings in a bit different way.

Anchor(convert)

Converting Mailbox Formats

Converting from MBX to MBOX

If you are using UW-IMAP and using the MBX format, you will need to convert it to MBOX format. The conversion process isn't pretty, but here is a script that works. You will need to get and compile the mailutil program from the UW-IMAP web site.

WARNING: Script versions listed here before 2006-05-18 11:40 UTC converted into mbx rather than mbox format, i. e. older versions did the opposite of what they were supposed to do. Use the version below:

# Written by Marc Perkel - public domain
# overhauled by Matthias Andree, 2006
# Usage: mbx-convert <filename>
# This code assumes there a user named "marc" with the primary group "marc".
# Change to any real user on your system.
# Yes - it look bizzare - but it gets the job done

# abort on error
set -e

user=marc
group=marc
homedir=/home/$user

if [ $# -ne 1 ] ; then
  echo >&2 "Usage: $0 <filename>"
  exit 1
fi

# set up automatic cleanup
trap 'rm -f "${homedir}"/in.$$ "${homedir}"/out.$$' 0

# First copy to users home dir and make the user the owner
cp "$1" "${homedir}/in.$$"
chown "$user":"$group" "${homedir}/in.$$"

# Run mailutil to convert as the user other than root
# mailutil requires this
su "$user" -c "mailutil copy in.$$ \#driver.unix/out.$$"

# create new file with same permissions/owner as old
cp -p "$1" "${1}.new"
# cat instead of copy leaves the original owner and permissions alone
if cat "${homedir}/out.$$" >"${1}.new" ; then
  # cat succeeded, rename file into place
  mv "${1}.new" "$1"
else
  # cat failed, remove temp file
  rm -f "${1}.new"
  exit 1
fi

Make a copy of some folders and test it first. Once you are satisfied that it works then:

  • Write a script to convert all your files.
  • Shut down your email system so files can't be modified
  • Copy all your email to a backup location in case you have to revert
  • Run the script
  • Turn on Dovecot and test to verify it is working
  • Important - make sure that you changed your SMTP configuration to write MBOX and not MBX.
  • Turn on SMTP and verify it is all working

User comments:

  • Is this hassle actually necessary? I have run mailutil as root like this (mailutil as provided by the PINE 4.61 package for SUSE Linux 10.0): mailutil copy /tmp/foo.mbx.orig '#driver.unix//tmp/foo.test' and did not encounter any problems. -- MatthiasAndree, 2006-05-18

Converting from MBOX to Maildir

[http://batleth.sapienti-sat.org/projects/mb2md/ mb2md] is a script which can convert the files for you.

Check also the User-Contributed Maildir Support section [http://www.qmail.org/top.html#usersoft on the qmail community site] for more choices.

Example (user's mail in ~someuser/mail and inbox in /var/mail/someuser):

cd ~someuser
mb2md-3.20.pl -s mail -R
mb2md-3.20.pl -m -d /var/mail/incoming/someuser
mv mail mail.old

Now the mail will be in ~someuser/Maildir.

If you are migrating from UW-IMAP, you also need to do:

cp -p .mailboxlist Maildir/subscriptions

Note that maildir handles subdirectories differently. You need to replace "/" with "." in the subscriptions file:

sed "s/\//\./" subscriptions > subscriptions.new
mv subscriptions.new subscriptions

Don't forget to update default_mail_env in dovecot.conf (as needed)

Converting from Maildir to MBOX

This is especially helpful if you want to archive your mail to a single file for storage on a CD, a PC, etc. But it can also be helpful if you want to use MBOX with dovecot.

Use the reformail program that comes with [http://www.courier-mta.org/maildrop/ maildrop]. You can also use the formail program that comes with [http://www.procmail.org/ procmail]. Here is a simple script showing how this works.

To use it, adjust the script to invoke the right command according to your system.

Then cd to the user's home directory (one level above Maildir) and run the script with two arguments: the mailbox name (You can use "." for the top-level folder), and the output mbox filename, for example:

cd ~hans
perl dw-maildirtombox.pl . >/tmp/hans-inbox
perl dw-maildirtombox.pl Sent >/tmp/hans-sent

WARNING: versions of the script provided here before 2006-05-18 11:40 UTC did not save the new/ directory contents, i. e. they missed unread messages!

# dw-maildirtombox.pl
# dw = Dovecot Wiki :-)

# NOTE! The output file must not contain single quotes (')!

# figure out which program to run
$cmd="reformail -f1";
system("$cmd </dev/null >/dev/null 2>/dev/null") == 0 or $cmd="formail";
system("$cmd </dev/null >/dev/null 2>/dev/null") == 0
or die "cannot find reformail or formail on your \$PATH!\nAborting";

$dir=$ARGV[0];
$outputfile=$ARGV[1];
if (($outputfile eq '') || ($dir eq ''))
{ die "Usage: ./archivemail.pl mailbox outputfile\nAborting"; }

if (!stat("Maildir/$dir/cur") || !stat("Maildir/$dir/new"))
{ die "Maildir/$dir is not a maildir.\nAborting"; }

@files = (<Maildir/$dir/cur/*>,<Maildir/$dir/new/*>);
foreach $file (@files) {
  next unless -f $file; # skip non-regular files
  next unless -s $file; # skip empty files
  next unless -r $file; # skip unreadable files
  $file =~ s/'/'"'"'/;  # escape ' (single quote)
  $run = "cat '$file' | $cmd >>'$outputfile'";
  system($run) == 0 or warn "cannot run \"$run\".";
}

None: Migration (last edited 2010-11-02 22:24:58 by TimoSirainen)