This documentation is for Dovecot v1.x, see wiki2 for v2.x documentation.
Differences between revisions 30 and 80 (spanning 50 versions)
Revision 30 as of 2005-12-02 22:35:53
Size: 9383
Comment:
Revision 80 as of 2010-11-02 22:24:58
Size: 6332
Editor: TimoSirainen
Comment: despam
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
[[TableOfContents]]
Line 5: Line 3:
Dovecot is one of the easiest IMAP servers to migrate to because of its very powerful configuration options. Dovecot can store email in both MBOX and MAILDIR formats making it compatible with most other mail storage formats. It is very flexible as to where it stores the email. Dovecot supports many methods for storing user data and password authentication. You can set it up to authenticate against multiple databases. '''WARNING: Badly done migration will cause your IMAP and/or POP3 clients to re-download all mails. Read this page carefully.'''
Line 7: Line 5:
Migration involves several separate tasks. You either need to convert your data or make Dovecot read your existing data. Dovecot is very good at being compatible and configurable so that it is likely to read your existing mail folders and user and password configurations. Tasks for conversion include: This page contains generic information related to migrating from another IMAP or POP3 server to Dovecot. You should read this page, and then look at the server-specific instructions:
Line 9: Line 7:
 * Where and how to store the email
 * User configuration - home directory structure - UID/GID - password storage
 * Authentication of users
 * [[Migration/UW|UW-IMAP / UW-POP3]]
 * [[Migration/Linuxconf|Linuxconf / VIMAP]]
 * [[Migration/Courier|Courier IMAP and POP3]]
 * [[Migration/Cyrus|Cyrus IMAP and POP3]]
 * [[Migration/Vm-pop3d|vm-pop3d]]
 * [[Migration/Teapop|teapop]]
 * [[Migration/BincIMAP|Binc IMAP]]
Dovecot is one of the easiest IMAP servers to migrate to because of its powerful configuration options. Dovecot can store email in both [[MailboxFormat/mbox|mbox]] and [[MailboxFormat/Maildir|Maildir]] formats, making it compatible with many existing servers. Dovecot is also very flexible as to where it stores the email. It supports many different databases for storing [[PasswordDatabase|passwords]] [[UserDatabase|user information]].
Line 13: Line 16:
When migrating from one IMAP server to another, you should make sure that these are preserved: Migration involves several separate tasks. You either need to convert your data or make Dovecot read your existing data. Dovecot is very good at being compatible and configurable, so it is likely to read your existing mailboxes and user and password configurations. Tasks for conversion include:
Line 15: Line 18:
 * Mailbox subscription list
  * User would be able to manually subscribe them again if you don't want to mess with it.
 * Message UIDs
  * If UIDs are lost, at the minimum clients' message cache gets cleaned
 * [[MailLocation|Where and how to store the email]]
 * [[PasswordDatabase|User authentication]]
 * [[UserDatabase|User database configuration]] (home directory structure, UID and GID)
== IMAP migration ==
When migrating mails from another IMAP server, you should make sure that these are preserved:

 1. Message flags
  * Lost flags can be really annoying, you most likely want to avoid it.
 1. Message UIDs and UIDVALIDITY value
  * If UIDs are lost, at the minimum clients' message cache gets cleaned and messages are re-downloaded as new.
Line 20: Line 29:
 * Message flags
  * Lost flags can be really annoying, you most likely want to avoid it.
 1. Mailbox subscription list
  * Users would be able to manually subscribe them again if you don't want to mess with it.
== POP3 migration ==
When migrating mails from another POP3 server, you should try to preserve the old UIDLs. If POP3 client is configured to keep mails in the server and the messages' UIDLs change, all the messages are downloaded again as new messages.
Line 23: Line 34:
Here's the more server-specific instructions: '''Don't trust the migration scripts or anything you see in this wiki. Verify manually that the UIDLs are correct before exposing real clients to Dovecot.''' You can do this by logging in using your old POP3 server, issuing UIDL command and saving the output. Then log in using Dovecot and save its UIDL output as well. Use e.g. {{{diff}}} command to verify that the lists are identical. Note that:
Line 25: Line 36:
= UW-IMAP =  * '''If a client already saw changed UIDLs and decided to start re-downloading mails, it's unlikely there is anything you can do to stop it. Even going back to your old server is unlikely to help at that point.'''
 * Some (many?) POP3 clients also require that the message ordering is preserved.
 * Some clients re-download all mails if you change the hostname in the client configuration. Be aware of this when testing.
Line 27: Line 40:
By default UW-IMAP allows access to whole home directory, and many people have chosen to store their mails in mail/ directory. This usually means that IMAP clients have set "IMAP namespace" to "mail/", which doesn't work well with Dovecot, as Dovecot by default uses mail/ directory directly. So if IMAP namespace is kept as "mail/", Dovecot would try to access "~/mail/mail/" directory. Some servers (UW, Cyrus) implementing both IMAP and POP3 protocols use the IMAP UID and UIDVALIDITY values for generating the POP3 UIDL values. To preserve the POP3 UIDL from such servers you'll need to preserve the IMAP UIDs and set {{{pop3_uidl_format}}} properly.
Line 29: Line 42:
So, removing the prefix from IMAP clients would be the first step. Next problem is that subscribed mailboxes are listed as "mail/box" or "~/mail/box" or "~user/mail/box" in subscriptions file. You'd have to remove the mail/ directory part from all of these. The subscriptions file name is also different, UW-IMAP uses `.mailboxlist` while Dovecot uses `.subscriptions`. If the server doesn't use IMAP UIDs for the POP3 UIDL, you'll need to figure out another way to do it. One way is to put the UIDL value into X-UIDL: header in the mails and set {{{pop3_reuse_xuidl=yes}}}. Some POP3 servers (QPopper) write the X-UIDL: header themselves, making the migration easy.
Line 31: Line 44:
Dovecot uses UW-IMAP compatible UID and message flag headers in mboxes, so that's not a problem.

Settings:

{{{
default_mail_env = mbox:~/mail:INBOX=/var/mail/%u
# make sure mbox_locks are the same with all software that accesses your mboxes
mbox_locks = dotlock fcntl
}}}

If you want to make a transparent migration to Dovecot without having to change the configuration on hundreds of client systems, you will need a slightly different configuration.

 * If your clients have the server prefix set to something like "~/mail", this will not work unless you enable mail_full_filesystem_access in your Dovecot configuration. Dovecot will otherwise reject mailbox names or prefixes that start with "~". Alternative to this with 1.0-tests is to [wiki:Namespaces set up multiple hidden namespaces].

 * You can either rename your ".mailboxlist" file to ".subscriptions" for all you users, or change the definition of SUBSCRIPTION_FILE_NAME in ''src/lib-storage/subscription-file/subscription-file.c''. In 1.0-test, this value is defined in ''src/lib-storage/subscription-fileindex/maildir/maildir-storage.h'' and ''src/lib-storage/subscription-fileindex/mbox/maildir-storage.h''.

 * If ~/mbox file exists, UW-IMAP moves mails from /var/mail/user there. Currently Dovecot doesn't support this feature, so you'll have to either move everyone's mails to ~/mbox and reconfigure MTA/LDA to store mails there, or alternatively get the ~/mbox users to move their mails back to /var/mail/user. This feature may be implemented later, but it's not planned in near future.

 * You should read [http://people.redhat.com/rkeech/maildir-migration.txt]. This document describes the migration process using scripts if you want to migrate from UW_IMAP with mbox to Dovecot using maildir (which will increase performance).

--- DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA ---

Only concerns Dovecot 0.99.x versions, 1.0-releases use them:

Every folder on a system using UW-IMAP has a system message in the top with the subject "DON'T DELETE THIS MESSAGE -- FOLDER INTERNAL DATA". This system message has information about the folder and its state. Dovecot 0.99.x does not use these system messages to anything so you can delete them without any problems - unless you have users using Pine together with an IMAP or POP3 mailclient.

= UW-POP3 =

By default Dovecot generates POP3 UIDs differently than UW-IMAP. If you want to preserve them, with 0.99.x you'll have to
[http://dovecot.org/patches/pop3-uidl-uwimap.patch patch Dovecot]. With 1.0-tests you can use:

{{{
pop3_uidl_format = %08Xv%08Xu
}}}

Note: Confirm that old and new UIDLs match. 0.99.x may not have the same uidl as UW-IMAP. But 1.0 appears to be so. Check with:

{{{
telnet localhost 110
user test
pass test
uidl
}}}

== VIMAP ==

VIMAP or [http://vimap.sourceforge.net/ Virtual Imap] was a patched version of WU-IMAP that was modified to work with [http://www.solucorp.qc.ca/linuxconf/ Linuxconf's] virtual email configuration. The project doesn't seem to have much life and [mailto:patrik.hall@multitronic.fi Patrick Hall] who wrote the patch is thrilled that Dovecot can now be configured to be fully compatible. Details of how to do this are described in the [wiki:VirtualUsers Virtual Users] section.

== Converting from MBX to MBOX ==

If you are using WU-IMAP and using the MBX format you will need to convert it to MBOX format. The conversion process isn't pretty, but here's a script that works. You will need to get and compile the mailutil program from the WU-IMAP web site.

{{{
#! /bin/bash

# Written by Marc Perkel - public domain
# Usage mbx-convert <filename>
# This code assumes a user named "marc". Change to any real user on your system.
# Yes - it look bizzare - but it gets the job done

# First copy to users home dir and make the user the owner
cp "$*" /home/marc/in
chown marc:marc /home/marc/in

# Run mailutil to convert as the user other than root
# mailutil requires this
su - marc -c "/usr/bin/mailutil copy in \#driver.mbx/out"

# cat instead of copy preserves the original owner and permissions
cat /home/marc/out > "$*"

# cleanup
rm /home/marc/in
rm /home/marc/out
}}}

Make a copy of some folders and test it first. Once you are satisfied that it works then:

 * Write a script to convert all your files.
 * Shut down your email system so files can't be modified
 * Copy all your email to a backup location in case you have to revert
 * Run the script
 * Turn on Dovecot and test to verify it is working
 * Important - make sure that you changed your SMTP configuration to write MBOX and not MBX.
 * Turn on SMTP and verify it is all working

= Courier =

Courier by default uses "INBOX." as private IMAP namespace, so it has exactly the same problems as described with UW-IMAP above. Again, if you want the migration to be transparent for IMAP clients see [wiki:Namespaces Namespaces] how to get namespaces to be Courier-compatible.

 * Courier's `courierimapsubscribed` is compatible with Dovecot's `subscriptions` file, just rename it and remove the "INBOX." prefixes. (NOTE: with 0.99.x it's `.subscriptions', with 1.0 it's `subscriptions' without the dot)
 * Courier's `courierimapuiddb` is compatible with Dovecot's `dovecot-uidlist` file, just rename it.
 * Courier's message flags are compatible with Dovecot (as it's specified by Maildir specification)
 * Courier's message keywords implementation isn't Dovecot compatible and there's currently no easy way to migrate them.

See also the [http://bendiken.net/scripts/ courier2dovecot] script by Arto Bendiken.

Settings:

{{{
# normal home directories
default_mail_env = maildir:~/Maildir
# for virtual users
default_mail_env = maildir:~/
}}}

For POP3 UIDL compatibility, use one of:

{{{
# Courier version 0 (using maildir filenames)
pop3_uidl_format = %f
# Courier version 1 (UID)
pop3_uidl_format = %u
# Courier version 2 (UIDVALIDITY and UID)
pop3_uidl_format = %v-%u
}}}

= Cyrus =

See [http://madness.at/projects/ cyrus2courier], it's Dovecot-compatible. Also mirrored at [http://dovecot.org/tools/ dovecot.org].

For POP3 UIDL compatibility, use one of:

{{{
# Cyrus (old format - up to v2.1.3)
pop3_uidl_format = %u
# Cyrus (new format - v2.1.4 and above)
pop3_uidl_format = %v.%u
}}}

= Other POP3 servers =

Some POP3 servers generate the X-UIDL header. To use this header when available and ease migration, use:

{{{
# Use X-UIDL header if available
pop3_reuse_xuidl = yes
}}}

Different POP3 servers generate UIDs differently. If you want to preserve them to avoid users downloading their mails twice, you'll need to figure out how the server generates the UID and patch Dovecot accordingly to do the same.
Some POP3 servers using Maildir uses the maildir base filename as the UIDL. You can use {{{pop3_uidl_format = %f}}} to do this.
Line 174: Line 48:
 * [http://www.openwall.com/popa3d/ popa3d] Generates MD5 sum from a couple of headers. Dovecot uses compatible MD5 sums internally but converts them into strings in a bit different way.  * UW-POP3: {{{pop3_uidl_format = %08Xv%08Xu}}}
 * qmail-pop3d: {{{pop3_uidl_format = %f}}}
 * Cyrus <= 2.1.3: {{{pop3_uidl_format = %u}}}
 * Cyrus >= 2.1.4: {{{pop3_uidl_format = %v.%u}}}
 * [[http://www.citadel.org|Citadel]] (all versions): {{{pop3_uidl_format = %u}}}
 * Dovecot 0.99: {{{pop3_uidl_format = %v.%u}}}
 * [[http://www.ex-parrot.com/~chris/tpop3d/|tpop3d]]:
  * Maildir: {{{pop3_uidl_format = %Mf}}} (MD5 sum of the maildir base filename in hex)
  * mbox: MD5 sum in hex of first 512 bytes of the message (or of the full message if it's less than 512 bytes).
 * [[http://www.openwall.com/popa3d/|popa3d]] Generates MD5 sum from a couple of headers. Dovecot uses compatible MD5 sums internally, but converts them into UIDL strings in a bit different way.
 * [[http://www.toontown.org/teapop/|teapop]] 0.3.8:
  * Maildir: {{{pop3_uidl_format = %Mf}}} (MD5 sum of the maildir base filename in hex)
  * mbox: MD5 sum of the message without the following lines: Status,X-Status,Lines and Content-Length.
 * Cucipop mbox: v1.31 uses its own homebrew checksum based on headers and body. Injection of X-UIDL: headers and pop3_reuse_xuidl=yes is the way to go.
 * qpopper: {{{pop3_reuse_xuidl=yes}}}

== IMAP <-> IMAP copying ==
If you don't care about preserving messages' UIDs, or if there isn't a simple way to preserve them, you can always migrate from another IMAP server to Dovecot by downloading the messages via IMAP from the old server and then uploading them to Dovecot via IMAP.

There are several different tools for this, for example [[http://www.washington.edu/imap/|UW-IMAP]]'s mailutil, [[http://freshmeat.net/projects/imapsync|imapsync]] and [[http://www.yippiemove.com|YippieMove]].

=== imapsync ===
Here's an example of how to run imapsync for a single user:

{{{
imapsync --syncinternaldates \
  --host1 192.168.1.57 --authmech1 LOGIN --user1 leah@example.com --password1 secret \
  --host2 127.0.0.1 --authmech2 LOGIN --user2 leah@example.com --password2 secret
}}}
It is quite easy to script this for a number of users, assuming you have their passwords. Even if you do not, imapsync also supports logging in as an admin user that has the ability to copy message for sub users, and a variety of other authentication options.

Migration to Dovecot

WARNING: Badly done migration will cause your IMAP and/or POP3 clients to re-download all mails. Read this page carefully.

This page contains generic information related to migrating from another IMAP or POP3 server to Dovecot. You should read this page, and then look at the server-specific instructions:

Dovecot is one of the easiest IMAP servers to migrate to because of its powerful configuration options. Dovecot can store email in both mbox and Maildir formats, making it compatible with many existing servers. Dovecot is also very flexible as to where it stores the email. It supports many different databases for storing passwords user information.

Migration involves several separate tasks. You either need to convert your data or make Dovecot read your existing data. Dovecot is very good at being compatible and configurable, so it is likely to read your existing mailboxes and user and password configurations. Tasks for conversion include:

IMAP migration

When migrating mails from another IMAP server, you should make sure that these are preserved:

  1. Message flags
    • Lost flags can be really annoying, you most likely want to avoid it.
  2. Message UIDs and UIDVALIDITY value
    • If UIDs are lost, at the minimum clients' message cache gets cleaned and messages are re-downloaded as new.
    • Some IMAP clients store metadata by assigning it to specific UID, if UIDs are changed these will be lost.
  3. Mailbox subscription list
    • Users would be able to manually subscribe them again if you don't want to mess with it.

POP3 migration

When migrating mails from another POP3 server, you should try to preserve the old UIDLs. If POP3 client is configured to keep mails in the server and the messages' UIDLs change, all the messages are downloaded again as new messages.

Don't trust the migration scripts or anything you see in this wiki. Verify manually that the UIDLs are correct before exposing real clients to Dovecot. You can do this by logging in using your old POP3 server, issuing UIDL command and saving the output. Then log in using Dovecot and save its UIDL output as well. Use e.g. diff command to verify that the lists are identical. Note that:

  • If a client already saw changed UIDLs and decided to start re-downloading mails, it's unlikely there is anything you can do to stop it. Even going back to your old server is unlikely to help at that point.

  • Some (many?) POP3 clients also require that the message ordering is preserved.
  • Some clients re-download all mails if you change the hostname in the client configuration. Be aware of this when testing.

Some servers (UW, Cyrus) implementing both IMAP and POP3 protocols use the IMAP UID and UIDVALIDITY values for generating the POP3 UIDL values. To preserve the POP3 UIDL from such servers you'll need to preserve the IMAP UIDs and set pop3_uidl_format properly.

If the server doesn't use IMAP UIDs for the POP3 UIDL, you'll need to figure out another way to do it. One way is to put the UIDL value into X-UIDL: header in the mails and set pop3_reuse_xuidl=yes. Some POP3 servers (QPopper) write the X-UIDL: header themselves, making the migration easy.

Some POP3 servers using Maildir uses the maildir base filename as the UIDL. You can use pop3_uidl_format = %f to do this.

Here is a list of POP3 servers and how they generate their UIDs. Please update if you know more:

  • UW-POP3: pop3_uidl_format = %08Xv%08Xu

  • qmail-pop3d: pop3_uidl_format = %f

  • Cyrus <= 2.1.3: pop3_uidl_format = %u

  • Cyrus >= 2.1.4: pop3_uidl_format = %v.%u

  • Citadel (all versions): pop3_uidl_format = %u

  • Dovecot 0.99: pop3_uidl_format = %v.%u

  • tpop3d:

    • Maildir: pop3_uidl_format = %Mf (MD5 sum of the maildir base filename in hex)

    • mbox: MD5 sum in hex of first 512 bytes of the message (or of the full message if it's less than 512 bytes).
  • popa3d Generates MD5 sum from a couple of headers. Dovecot uses compatible MD5 sums internally, but converts them into UIDL strings in a bit different way.

  • teapop 0.3.8:

    • Maildir: pop3_uidl_format = %Mf (MD5 sum of the maildir base filename in hex)

    • mbox: MD5 sum of the message without the following lines: Status,X-Status,Lines and Content-Length.
  • Cucipop mbox: v1.31 uses its own homebrew checksum based on headers and body. Injection of X-UIDL: headers and pop3_reuse_xuidl=yes is the way to go.
  • qpopper: pop3_reuse_xuidl=yes

IMAP <-> IMAP copying

If you don't care about preserving messages' UIDs, or if there isn't a simple way to preserve them, you can always migrate from another IMAP server to Dovecot by downloading the messages via IMAP from the old server and then uploading them to Dovecot via IMAP.

There are several different tools for this, for example UW-IMAP's mailutil, imapsync and YippieMove.

imapsync

Here's an example of how to run imapsync for a single user:

imapsync --syncinternaldates \
  --host1 192.168.1.57 --authmech1 LOGIN --user1 leah@example.com --password1 secret \
  --host2 127.0.0.1    --authmech2 LOGIN --user2 leah@example.com --password2 secret

It is quite easy to script this for a number of users, assuming you have their passwords. Even if you do not, imapsync also supports logging in as an admin user that has the ability to copy message for sub users, and a variety of other authentication options.

None: Migration (last edited 2010-11-02 22:24:58 by TimoSirainen)