This documentation is for Dovecot v1.x, see wiki2 for v2.x documentation.
Differences between revisions 65 and 68 (spanning 3 versions)
Revision 65 as of 2008-05-08 04:48:11
Size: 8408
Editor: cm74
Comment:
Revision 68 as of 2008-08-03 20:26:32
Size: 5177
Editor: 84
Comment: Oop, got them mixed up
Deletions are marked like this. Additions are marked like this.
Line 40: Line 40:
 * qmail-pop3d: {{{pop3_uidl_format = %f}}}
Line 65: Line 66:

''''''

'''TESTED AND RUNNING, MIGRATING from MBOX to MAILDIR, PRESERVE UID and OUTLOOK sees the email as downloaded. I hope this helps a lot of people out there. :) Feel free to post this somewhere else too.'''

-----
Horn Wijaya ( hwijaya@nodens.biz ) 8th May 2008

I have been searching for weeks for a foolproof method to migrate all my users mbox to maildir/dovecot without having the pain of downloading all emails again, especially for heavy users. It would cause frustration to thousands of my users.

So.... eventually I found out why Outlook will keep downloading the emails despite all the configuration done in the dovecot.conf. Apparently the UID is generated realtime. When it first generated, it took the time the process runs.

Before I migrate, here's the email sample which I have downloaded to my outlook (and keep email on server), and then I use telnet to pop to get the uidl list.
+OK POP3 localhost.localdomain v2003.83rh server ready
user *****
+OK User name accepted, password please
pass *****
+OK Mailbox open, 444 messages
uidl
+OK Unique-ID listing follows
1 47b1303d00004c2d
2 47b1303d00004c2e
3 47b1303d00004c2f
...

47b1303d = 1202794557 = 12 Feb 2008 13:35:57 the date and time the mbox was created 00004c2d = 19501 the IMAP counter so far.... sequently added as you could see with email #2 and email #3 Then I switch my POP server to dovecot, and migrate my MBOX to Maildir, and I do a telnet to get the UIDL +OK Dovecot ready. user ***** +OK pass ***** +OK Logged in. uidl +OK 1 4822815000000001 2 4822815000000002 3 4822815000000003 ...

48228150 = 1210220880 = NOW 00000001 = counter starts at 1 No wonder outlook sees the emails are NOT-DOWNLOADED-YET status.

Then I ponder for another few days and the key is in the dovecot-uidlist...... How to fill it with the correct info.... and there's a way. Which I hope could be implemented in the next version of the converter. Here's the step I do to reach my objective which is '''to get UID 47b1303d00004c2d in the first email''':

1. mb2md-3.20.pl -s /var/mail/hwijaya -d home/hwijaya/Maildir/, that's converting 444 emails from mbox to 444 files in the Maildir/cur folder

2. cd /home/hwijaya/Maildir/cur

3. grep IMAPbase *

     and I get

     1210219860.000000.mbox:2,S:X-IMAPbase: '''1202794557 19944''' The bolded strings are the KEY to the ORIGINAL UID

4. So in order to prepare the environment to fool it, some calculations are needed:

     How do we get 19501 as the first counter? as expected in the first POP TEST above

''' 19944 - 444 emails + 1 = 19501'''

5. cd /home/hwijaya/Maildir

6. echo '3 V'''1202794557''' N'''19501'''' > dovecot-uidlist

7. DONE....... pop it via outlook and nothing will be downloaded, after pop or imap is processed, the dovecot-uidlist is auto-filled, index auto generated.

Proof:

+OK Dovecot ready.
user *****
+OK
pass *****
+OK Logged in.
uidl
+OK
1 47b1303d00004c2d
2 47b1303d00004c2e
3 47b1303d00004c2f

I assume the same method could go into individual folders in the imap (under .mailboxlist or .subscriptions)

Please do let us know if there's any cases/environment that it won't work.

Thanks very much, and a big migration job is in front of me now... :)

Migration to Dovecot

This page contains generic information related to migrating from another IMAP or POP3 server to Dovecot. You should read this page, and then look at the server-specific instructions:

  • [:Migration/UW:UW-IMAP / UW-POP3]

  • [:Migration/Linuxconf:Linuxconf / VIMAP]

  • [:Migration/Courier:Courier IMAP and POP3]

  • [:Migration/Cyrus:Cyrus IMAP and POP3]

  • [:Migration/Vm-pop3d:vm-pop3d]
  • [:Migration/Teapop:teapop]

  • [:Migration/BincIMAP:Binc IMAP]

Dovecot is one of the easiest IMAP servers to migrate to because of its powerful configuration options. Dovecot can store email in both [:MailboxFormat/mbox:mbox] and [:MailboxFormat/Maildir:Maildir] formats, making it compatible with many existing servers. Dovecot is also very flexible as to where it stores the email. It supports many different databases for storing [:PasswordDatabase:passwords] [:UserDatabase:user information].

Migration involves several separate tasks. You either need to convert your data or make Dovecot read your existing data. Dovecot is very good at being compatible and configurable, so it is likely to read your existing mailboxes and user and password configurations. Tasks for conversion include:

  • [:MailLocation:Where and how to store the email]

  • [:PasswordDatabase:User authentication]

  • [:UserDatabase:User database configuration] (home directory structure, UID and GID)

IMAP migration

When migrating mails from another IMAP server, you should make sure that these are preserved:

  1. Message flags
    • Lost flags can be really annoying, you most likely want to avoid it.
  2. Message UIDs and UIDVALIDITY value
    • If UIDs are lost, at the minimum clients' message cache gets cleaned and messages are re-downloaded as new.
    • Some IMAP clients store metadata by assigning it to specific UID, if UIDs are changed these will be lost.
  3. Mailbox subscription list
    • Users would be able to manually subscribe them again if you don't want to mess with it.

POP3 migration

When migrating mails from another POP3 server, you should try to preserve the old UIDLs. If POP3 client is configured to keep mails in the server and the messages' UIDLs change, all the messages are downloaded again as new messages.

Some servers (UW, Cyrus) implementing both IMAP and POP3 protocols use the IMAP UID and UIDVALIDITY values for generating the POP3 UIDL values. To preserve the POP3 UIDL from such servers you'll need to preserve the IMAP UIDs and set pop3_uidl_format properly.

If the server doesn't use IMAP UIDs for the POP3 UIDL, you'll need to figure out another way to do it. One way is to put the UIDL value into X-UIDL: header in the mails and set pop3_reuse_xuidl=yes. Some POP3 servers (QPopper) write the X-UIDL: header themselves, making the migration easy.

Some POP3 servers using Maildir uses the maildir base filename as the UIDL. You can use pop3_uidl_format = %f to do this.

Here is a list of POP3 servers and how they generate their UIDs. Please update if you know more:

  • UW-POP3: pop3_uidl_format = %08Xv%08Xu

  • qmail-pop3d: pop3_uidl_format = %f

  • Cyrus <= 2.1.3: pop3_uidl_format = %u

  • Cyrus >= 2.1.4: pop3_uidl_format = %v.%u

  • [http://www.citadel.org Citadel] (all versions): pop3_uidl_format = %u

  • Dovecot 0.99: pop3_uidl_format = %v.%u

  • [http://www.ex-parrot.com/~chris/tpop3d/ tpop3d]:

    • Maildir: pop3_uidl_format = %Mf (MD5 sum of the maildir base filename in hex)

    • mbox: MD5 sum in hex of first 512 bytes of the message (or of the full message if it's less than 512 bytes).
  • [http://www.openwall.com/popa3d/ popa3d] Generates MD5 sum from a couple of headers. Dovecot uses compatible MD5 sums internally, but converts them into UIDL strings in a bit different way.

  • [http://www.toontown.org/teapop/ teapop] 0.3.8: Maildir: pop3_uidl_format = %Mf (MD5 sum of the maildir base filename in hex)

    • Maildir: pop3_uidl_format = %Mf (MD5 sum of the maildir base filename in hex)
    • mbox: MD5 sum of the message without the following lines: Status,X-Status,Lines and Content-Length.

IMAP <-> IMAP copying

If you don't care about preserving messages' UIDs, or if there isn't a simple way to preserve them, you can always migrate from another IMAP server to Dovecot by downloading the messages via IMAP from the old server and then uploading them to Dovecot via IMAP.

There are several different tools for this, for example [http://www.washington.edu/imap/ UW-IMAP]'s mailutil and [http://freshmeat.net/projects/imapsync imapsync].

imapsync

Here's an example of how to run imapsync for a single user:

imapsync --syncinternaldates \
  --host1 192.168.1.57 --authmech1 LOGIN --user1 leah@example.com --password1 secret \
  --host2 127.0.0.1    --authmech2 LOGIN --user2 leah@example.com --password2 secret

It is quite easy to script this for a number of users, assuming you have their passwords. Even if you do not, imapsync also supports logging in as an admin user that has the ability to copy message for sub users, and a variety of other authentication options.

None: Migration (last edited 2010-11-02 22:24:58 by TimoSirainen)