New Program: Merge Thunderbird / Evolution style (mbox) folders
To keep my desktop and my laptop in sync with each other, I use a program called Unison, and it works great for almost everything I throw at it. It’s fast, smart, and has the right amount of control and options to do exactly what I want, except for when it comes to one thing: mail.
Unfortunately, most of the time Unison’s diff-style merge doesn’t work for mbox mail files, which is what both Evolution and Mozilla SeaMonkey / Thunderbird (as well as a lot of other Email clients) use. It’s about impossible to merge two mail folders by hand if you have even more than 50 or so messages, and I didn’t know any way to do it using existing tools. So to fix this, I wrote a small command-line Ruby program that knows how to merge two mail folders without duplicates. Once I finished though, I had no idea if my program wasn’t losing mail messages in the merge, so I wrote another program that displays the differences between two mail folders in terms of messages.
- For OS X / Linux: mboxmerge.rb, mboxdiff.rb
- For Windows (pre-compiled using RubyScript2Exe): mboxmerge.exe, mboxdiff.exe
Some examples:
mboxmerge.exe Inbox_A Inbox_B Inbox_C > New_Inbox
mboxdiff.exe Inbox_A Inbox_B
cat `find ~/mailboxes | grep ‘^C’` | mboxmerge.rb > C_Mailboxes
That last one was getting a little bit fancy, but you get the idea. The program uses the From, Subject, To, the Date, and the message text to determine if a message is unique, so it’s pretty thorough at telling which messages are identical between the mail files, so it’s safe to use it for synchronization purposes (which is why I wrote it anyways!)
Now, I use this on my own files and I’m reasonably sure that it won’t chew up your mail, but please make sure that you have a backup in case things go wrong or my program randomly freaks out and deletes everything. It will never write to its input files, so there’s no chance of that failing but the standard disclaimer applies.
Update: Also check out the script to automate syncing the local and remote copies of a mail folder
This appeared to be just what I was looking for! However, I may be missing something. Here’s what I’ve tried. I successfully merge two mail files using mboxmerge.rb, then replace the original mail folder with the merged folder while Evolution is not running.
I then launch Evolution and attempt some operation within the merged folder, say expunge some email files. I then get an error message:
Error while Expunging folder.
Summary and folder mismatch, even after a sync
My system: Fedora 5, Evolution 2.6.3.
Jim Mehl
20 May 07 at 12:50 pm
This is some sort of Evolution bug – Evo keeps an index of your Email files so it can quickly go through them but for some reason, it gets confused. The fix is to go into your Evolution directory and delete some of those .index and .meta files (back it up first to make sure nothing weird happens!)
Paul Betts
21 May 07 at 12:08 pm
Hi Paul, I tryed you mboxmerge.exe, the script worked fine for me for single folder made with Thunderbird. When I try to export my inbox with 100 sub-folders I have to merge manualy every folder. I export only mail Inbox1 (~144MBsize) and try to make mboxmerge.exe with other Inbox2 (~120MB size), but here I recieved:
C:\1>mboxmerge inbox1 inbox2 > inbox
” has no date!
C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge.rb:49:in `mk
time’: no implicit conversion from nil to integer (TypeError)
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:49:in `parsedate_to_time’
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:77:in `parse’
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:65:in `each’
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:65:in `parse’
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:119
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:117:in `each’
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:117
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:115:in `each’
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\app\mboxmerge
.rb:115
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\bootstrap.rb:
50:in `load’
from C:\DOCUME~1\Mobile1\LOCALS~1\Temp\eee.mboxmerge.exe.8\bootstrap.rb:
50
So, I do not know where is the problem. May I have more info to do it?
My situation is: I have 2 PC with 2Thunderbirds. Every mail client open same mail account. I make around 100 sub folders in Inbox. I whan to merge this 2 mail box from Thunderbirds.
Best Regards
Anatoliy Trifonov
11 May 08 at 9:51 pm
@Anatoliy: I believe I’ve fixed this problem in the Ruby version of this script, try using that instead of the EXE (let me know if you have trouble!)
Paul Betts
11 May 08 at 11:19 pm
Say, how long does you script take to merge 2 mailboxes, each having about 3000 mails with about 90% of them duplicates?
oliver
17 Sep 08 at 8:20 am
@oliver – No clue, but it shouldn’t take *too* long to do. Give it a try
Paul Betts
18 Sep 08 at 1:02 pm
I tried your programs on a Thunderbird mailbox folder where my wife and I have collected different subsets of the total set of messages.
I found that this folder exposes two problems in your code:
1. One message in this mailbox contains the following header lines:
>>>>>>>> Start lines <<<<<<<>>>>>>> End lines <<<<<<<>>>>>>> Start lines <<<<<<<<
—– Forwarded by Lars Greninger/Austin/IBM on 02/13/2009 10:04 PM —–
From:
Good Friend
To:
Lars Greninger/Austin/IBM@IBMUS
Date:
02/13/2009 06:36 PM
Subject:
Re: IBM neutron bomb
>>>>>>>> End lines <<<<<<<<
No idea where the extra EOLs came from (Lotus Notes probably), but these data lines caused an exception when your code saw the “Date:\n” data line as a new header line.
This exception was due to your not chomp!-ing the string ’s’ before testing for a zero length so all body lines were processed as header lines.
Fixing these two problem areas gave me an excellent reason to learn a little Ruby. Neat language! And very useful programs!
Lars Greninger
14 Apr 09 at 9:58 am
Whoops. My first defect example and fix got swallowed by the blog s/w.
The defect was that header line keywords are case-insensitive but you require them to start with a capital. I had a message that used “date”, etc. Easily fixed by tweaking the regexps to accept, for example, /^[D|d]date/ instead of /^Date/.
Lars Greninger
14 Apr 09 at 10:03 am
Should we be worried about these .msf files in Thunderbird? This program seems to work, which is a good thing since I want to merge my accounts from separate Linux distros I have installed.
Dallis Brockway
15 Apr 09 at 11:57 pm
@Dallis – afaik, the msf files are just indexes, so as long as the Mbox file itself is safe, everything should be fine. I don’t guarantee it though, as I don’t really use Thunderbird
Paul Betts
22 Apr 09 at 12:05 am
My situation is this:
I installed thunderbird on my linux system and it worked fine for few months but after that I had to change my system. And on new system i again configured thundered but what strange thing I noticed it was not downloading all the previous mails aand also those mails were not on my yahoo account.Now I m in trouble !!!!!!!
Can you suggest any means that I can merge .thunderbird folder of previous system and my new system in a synchronised way……..
Please help me!!!
I am totally lost because older mails are very important,.
If anyone has suggestions please help me
Akthtar
11 Sep 09 at 12:52 am
mboxmerge just made my day after using a different comp for seven months. Got all my Sent folders synced without a single issue.
Great work Paul!
Dominique
Dominique Stender
8 Dec 09 at 12:55 pm
[...] With a bit of research I found that I was not the only person with this problem and even better I found the blog by Paul Betts containing an article "Merge Thunderbird / Evolution style (mbox) folders". [...]
Syncing / merging Thunderbird mailbox files | Dominique Stender
14 Dec 09 at 1:19 am
[...] conviene que generes un perfil adicional para resguardarlos o puedes intentar combinarlos con Mboxmerge, creando antes una copia de [...]
Bitelia Responde: Sincronizar Thunderbird | Bitelia
25 Dec 09 at 6:03 am