Moving 12 years of email from GMail to FastMail

In 2013, when it became clear, primarily through Edward Snowden’s heroic actions, that the level of snooping by the US and other governments was far greater than any of us would have thought, I moved all of my data out of the US and of course blogged about it (that blog post has been read almost 70000 times; I think for many people this is an important issue).

This included migrating 60000 emails away from my beloved GMail (I got my GMail invite from The Vogon Poet on August 24, 2004. At that time, you could only get GMail by invitation. It was pretty exciting! (I have emails from before 2004, back to ’93 or ’94, but those are in a backup archive somewhere.)) to the little Synology DS213j standing next to my desk at the time.  This was all well and good behind the stable Dutch 100 Mbit/s down / 10 Mbit/s up cable connection I had, but when we decided to move back to South Africa, where home internet is a few years behind The Netherlands, I ended up having to pay for a virtual private server in Cape Town (to keep latency between me and my mail server manageable) and having to admin my own dovecot IMAP and postfix SMTP server.

Initially this was workable, until the Nth time that I had to interrupt my real job (which has nothing to do with mail servers) to apply a security patch or get the VPS booting again after a botched kernel upgrade. Besides that, I had to deal with keeping my server out of over-enthusiastic spam blacklists and whatnot. Also, inspite of mu4e, I did end up missing the fast graphical GMail web interface.

So, it was with a great deal of tail between my legs that on June 10, 2015 (I have a lab journal, remember) I went right back back to GMail. My mail setup, although pleasingly decentralised, was costing me too much time and hence actual money.

Fast forward to July 15, 2016 (there’s that lab journal again…) when, after receiving an email from Google asking me to indicate how exactly I would like them to use my data to customise adverts around the web, and after thinking for a bit about what kind of machine learning tricks I would be able to pull on you with 12 years of your email, I decided that I really had to make alternative plans for my little email empire.

Somehow FastMail came up and in one of those impulsive LET’S WASTE SOME TIME manoeuvres, I pressed the big red MIGRATE button!

The rest of this post is my mini-review of the FastMail service after almost 3 weeks of intensive use.

Importing mail from GMail

The main import & export window
The main import & export window
IMAP migration configuration dialog
IMAP migration configuration dialog

The Settings | Import & Export option in FastMail was easy to setup. It knows how to authenticate with GMail, even when you make use of two-factor authentication, like I do and you probably should.

The import takes place via the GMail IMAP interface. It’s important to remember that via the IMAP client, an email tagged in GMail with both important and info will appear in two different folders. Because of this, I did check the no duplicates checkbox, but still I noticed that my 15 GB FastMail evaluation mailbox was filling up more quickly than I would have expected.

After a support request which was responded to within minutes (bonus), I discovered the Quota Usage screen and could see that the duplicate detection did indeed not seem to work correctly during the import. Based on more tips from the support tech, I made use of the Mass delete or remove duplicates module (Settings | Folders | Scroll all the way to the bottom of the page) to delete thousands of duplicate emails during the import. This was indeed because of emails appearing in multiple IMAP folders due to their GMail tags. Note: Friend and reader stefanvdwalt reported the exact same mail duplication during import issue which in his case did go over quota, so do keep an eye on this!

After a day or so (during which I could email more or less normally) I received an import report from FastMail claiming that the import had been successful, except for this error:

Log: Fri Jul 15 17:49:17 2016; cpbotha/; Migrating folder Inbox -> Inbox
Log: Fri Jul 15 17:49:17 2016; cpbotha/; Creating local folder Inbox
Log: Fri Jul 15 17:49:17 2016; cpbotha/; Error migrating remote folder Inbox: Failed to create Inbox. IMAP Command : 'create' failed. Response was : no - Mailbox already exists

The import had managed to figure out that GMail Sent should map to FastMail Sent for example, but Inbox was probably too special to map in the same way. I fixed this by firing up my trusty Thunderbird, and using IMAP to drag and drop emails from my GMail Inbox to my shiny new FastMail Inbox.

In retrospect, I should have selected Create under new sub-folder in the IMAP migration configuration instead of Merge into existing folders. I discovered later that moving thousands of emails to a different folder is near instantaneous in the FastMail web-app.

What I like

Webmail speed

I live more or less at the southern tip of the African continent. My lowest latency connection with the rest of the internet is via undersea optic cable to Europe (about 140ms ping).

The FastMail web servers are in the USA, which is, as the ping flies, much further away. I was not expecting much from the webmail, but colour me surprised when I discovered that this felt subjectively faster than GMail (who have servers everywhere, even down here). Things remained snappy, even with all 50000 of my conversations imported.

As far as I can figure out, it seems that much of this is due to FastMail’s self-designed but open source IMAP-replacement called JMAP. JMAP has been designed for low latency, and for improved battery life. What it does differently, is batch requests together, and it also has optimisations specifically for interactive webmail.

The web-app has full support for keyboard shortcuts, which increases the subjective perception of speed.

Webmail search

For my purposes, search in FastMail is on par with that of GMail. I can dig up any of my emails, back up to 2004, in seconds.

FastMail advanced search interface
FastMail advanced search interface

What’s also very useful, is that you can turn any search into a virtual folder.

Tech support

This is one area where Google really can’t hold a candle to FastMail. If something goes wrong with your gmail account (this hardly ever happens, but it’s possible) it’s almost impossible to get hold of any kind of official tech support. Here’s a recent story where a GMail user’s account was summarily terminated. There was probably some kind of ToS infringment, but the user has no idea what or why, and has lost all access to their emails and contacts database.

So far I’ve contacted FastMail tech support twice: Once during my email migration, and once to confirm the absence of the “quote selected text in reply” feature (discussed below). In both cases, I was helped by real humans who responded very quickly and courteously to my support requests.

Email and contacts (and calendar) out of Google’s view

I’m still of the opinion that Google makes fantastic and valuable products. However, with all of their data mining know-how and resources, one has to decide how much of one’s personal information one is willing to trade in for the use of these fantastic products.

With FastMail, I have been able to extricate my significant email archive (2004 to 2016, 50000 conversations) as well as my contacts database. I’m still making use of Google Calendar, because of bunches of sharing going on with family members, but I have the option of moving that out also.

By the way, the FastMail Calendar web interface is more than capable (and pretty enough) to replace Google’s version.

What I don’t like

Missing integrations: Todoist

GMail, being as popular as it is, has tonnes of integrations with other apps. In my case, I will really miss the Todoist for Gmail extension. With this, I had a mini-todoist window inside my GMail, and I could turn any email into a task at the click of a button (or the press of a shortcut).

Because FastMail email URLs seem to be persistent, I use the Todoist Chrome extension’s “Add to Todoist” context menu action to add the URL and email subject as a task. This not as nice as the gmail-specific extension (the task goes immediately into the todoist inbox, without the possibility to edit metadata such as due date and tags).

Missing feature: Quote selection in reply

In Gmail and in Thunderbird, if you select text in an incoming email and then reply, that selected text is quoted in the reply email. Unfortunately, this feature is not available in the FastMail web-app, and they have no plans to implement it.

I use both the FastMail web-interface as well as Thunderbird, because of its great PGP email encryption and signature support (hey, find me on keybase, send me encrypted email!), so this issue is somewhat ameliorated. Still, it would have been nice.

Android app lag

I do have FastMail’s Android app on my telephone. The app is a Cordova / PhoneGap / CrossWalk style unit with real-time email push and notification via Google Cloud Messaging (this is a relatively energy-efficient way for android phones to get push notification and is natively supported by FastMail).

However, there is a few second lag when I open the inbox, so I prefer using the pro version of AquaMail, a great Android IMAP mail client. I have this set to 15 minute polling for new email, as IMAP IDLE (push, in other words) is not as battery efficient as GCM or Apple’s email push. Opening any folder or email in AquaMail is of course instantaneous, as the emails live on the phone.

That being said, I use the FastMail app for searching, which is just as fast and as effective as the web-app.

THAT being said, FastMail really needs to implement some sort of caching in the Android app for lightning fast folder and email access. (The FastMail app is quite attractive, I would prefer using it more.)

FastMail Android app Calendar screen, from the Google Play page.
FastMail Android app Calendar screen, from the Google Play page.

Niggle: Creating an email alias / incoming route automatically creates a new sending identity

FastMail can manage the DNS for any of the custom domains that you assign to it, which is super useful if you don’t already have a DNS service.

I already make use of webfaction’s DNS for all of my domains, so I chose to add DNS records to designate fastmail as the official MX for those domains. (All of this is explained clearly in the FastMail help.)

When you do this, you have to create an email alias for each incoming address you would like to receive mail for (you can also create a catchall, but this could result in more spam arriving in your inbox). For each and every alias, FastMail automatically creates an outgoing (from address) identity. While this is usually quite convenient, I have quite a number of incoming addresses, but I only ever send from a subset of these addresses, so the drop-down list with sending identities became quite unwieldy.

I deleted all of the unnecessary identities. What would help, would be if FastMail were to implement most-used-at-the-top sorting for that drop-down.

Other noteworthy points

Domain setup

For my most important domains, I have set FastMail to be the MX. I have also performed the necessary SPF and DKIM setup: FastMail gives super useful feedback in its configuration screens to help you with this. For these domains, I send mail directly via the FastMail SMTP servers, and mail is delivered directly to FastMail servers. Nice and simple.

Domain setup feedback screen.
Domain setup feedback screen.

For some other email accounts I have with clients, FastMail supports POP fetch from and SMTP send via foreign servers.

iOS Push support

If you use any Apple iOS devices to read your mail, you’ll be pleased to know that FastMail, with help from the big A, fully supports iOS push. This means battery efficient real-time incoming emails to make it even more difficult for you to focus on That One Really Important Thing.

Android contact syncing with CardDAV

With google contacts, syncing on Android just works, and it works really well. To sync my contacts with FastMail’s Address Book instead, I bought the pro version of the CardDAV android app for 24 South African Ront (that’s about EUR 1.5). This works as a sync provider, so once setup, the process is also pretty much transparent.

Final thoughts

So there you have it: A hopefully helpful story, with included mini-review, about my move from GMail to the FastMail service.

So far, my conclusion is that this is a service that is technically more than capable of replacing GMail, even for power users. Furthermore, FastMail’s primary (and in fact only) business model is to charge you money for making sure that you can keep on emailing like a boss. Together, this makes for an offer that I could not refuse.

P.S. Let me know in the comments if you would like me to add anything else to this post.

P.P.S. You can also join the lively Hacker News discussion of this post!

Dear USA, my data has left your building.

NSA, GCHQ, Prism, FISA, Project Bullrun, Sigint.

After Edward Snowden, former CIA and NSA employee, started revealing how massively, intensely and easily we are all being spied upon by the intelligence agencies of various governments, the terms above have suddenly been spending a great deal more time in the media.


It turns out that government agencies are allowed to extract, at a whim, your and my data from service providers, such as Google, Microsoft and Yahoo. There is no real legal process (unless you can call a secret judge in a secret court giving a secret order a real legal process), especially if you’re not a US citizen, and the providers that have been forced to give up your data in this way are not allowed to notify you about your digital self being violated. So even if they say that you shouldn’t worry, you can never be entirely sure.

Furthermore, it has also been revealed that the NSA has for years being acquiring encryption keys via legal (secretly forcing companies to give them the keys) and extra-legal (simply hacking into company servers) means. Even worse, they have for years been deliberately introducing security weaknesses into software products and encryption software in order to be able to crack open your data even more easily.

You can read more about this state of affairs in The Guardian’s NSA files. The Guardian has been doing a sterling job of analysing and bringing to light the depths to which our governments have sunk. There’s a whole lot of information, and most of it is quite upsetting.

For me the final straw was when secure email service lavabit voluntarily shut itself down, when faced with the prospect of being forced to leak user information to the US government without being allowed to tell anyone. The message on the site is quite chilling, and concludes with the following:

This experience has taught me one very important lesson: without congressional action or a strong judicial precedent, I would _strongly_ recommend against anyone trusting their private data to a company with physical ties to the United States.

At this point, I was a super happy and pretty heavy user of a number of US-based services, including GMail (all my email, about 40000 conversations consisting of 60000 mails, that’s excluding my work email which I also hosted on GMail), Google+ Photos (all my photos, about 21000 of ’em), Google Drive, Dropbox (50G of data spread out over 120000 files). In all cases, I still consider these to be best of class services. In putting my money where my mouth is, I was paying both Google and Dropbox for extra storage.

I also had no problem with Google filtering through my email to show me targeted advertising. This is the deal I had with them. I also had no problem with the possibility of someone getting my data after due legal process. However, the idea that some NSA or other government agency flunky could quite easily stick their grubby paws into my data, and that I would never know about this, was too much.

There’s probably nothing much of interest in my data. However, it has become a matter of principle; Privacy is a basic human right. Here’s an old essay by Bruce Schneier if you need to read more about why privacy is so important.

In short: It was time to extricate all of my lovely data from probably well-meaning US companies, thanks to the ridiculously powerful and secretive NSA, and thanks to all of its shadowy counterparts around the world.

Here’s how I did it:

  • Considered building another low-cost Linux server, or even a Raspberry Pi. Decided against this due to time required for configuration and acquired a Synology DS213j NAS, which is at this moment standing on the desk about 1 metre to my left. My recommendation: Just get this, you won’t be sorry.
  • Downloaded 60000 emails to Synology using Thunderbird mail client. Deleted everything from GMail. Google engineers assure me that after a few months, data will really be gone.
  • My webhoster (WebFaction) receives mail for all my domains. My Synology retrieves mail every 5 minutes via POP (you can set this up via Roundcube on the Synology) and deletes it from WebFaction.
  • Outgoing mail is relayed by the Synology via the WebFaction SMTP server. I don’t have to worry too much about blacklisting and whatnot, my hoster does this.
  • I’m back to interacting with my mail using Thunderbird and IMAP SSL. The loss of GMail conversation view was initially really REALLY painful. People have forgotten the ancient art of quoting. However, I’ve configured Thunderbird to archive all mail to year-stamped archive folders, and to put my sent mail there. Poor-man’s Conversation View! (the conversations plugin is wonky. it’s shocking how much the availability of GMail, which works really well, has stunted the development of alternative email clients) Importantly, I am now able to use OpenPGP again for the strong encryption and cryptographic signing of my emails.
  • On my Android telephone (whoops…) I am using the Kaiten IMAP client.
  • All the data I had in Dropbox is now being synced between the Synology, two laptops and a workstation using BitTorrent Sync. This peer-to-peer syncing system is still a little rough around the edges, but falls squarely in the category of “Best Things Since Sliced Bread”, and it’s FAST. CloudStation, Synology’s dropbox-inspired solution, was just far too slow on my Synology model.
  • My photos (21000 of them) have been downloaded from Google+ Photos (thank you Google Takeout) and are now being served from the Synology using PhotoStation.
  • My music (5400+ tracks) is downloading from Google Music as we speak, and will be served from the Synology using AudioStation.
  • I make incremental backups of everything to an encrypted external USB drive, using dirvish. I will probably add an extra external drive to the mix and try to keep that off site.

It’s been an interesting process moving my stuff out, and getting used to these alternative systems is sometimes slightly uncomfortable, but I am quite happy with the end result. I hope that more people will take this step, and I really hope that more and easier-to-use alternatives for secure email (such as mailpile) and for ubiquitous private data will become available.

Addendum 2013-09-16

My submission of this post spent some time on the Hacker News front page, and from there was picked up by reddit as well. This brought many comments, a number of which were positive and thoughtful, and a number of which that were far less so. It’s amazing how anonymity and comment sections can bring out the worst in people. (if you have to know, the Hacker News community is generally MILES more polite than reddit)

In any case, I wanted to clarify an issue or two: After moving my data away from GMail and Dropbox, I am not under any impression that my data is now secure. I can still be hacked. My hardware and software could be full of backdoors. My email will still be read as it jumps from server to server, probably ending up in someone else’s GMail. :) However, if more people were to move their data out to their own premises, it becomes more complicated and costly for government agencies to monitor us all. At the moment, the NSA cuts deals with a few large email and other cloud service providers, and with that they’re able to monitor large swathes of users. However, if more of those users were to move away, many more deals have to be cut and servers hacked, costing more time and more money. Add to that increased used of OpenPGP (which I do use, and mention in my post), and it becomes even more difficult. I know that I’m just a drop in a bucket, but hey, at least I am a drop in a bucket!

My goal with posting this was to show that it’s relatively easy to move much of your data away. I have the feeling that many of the most impolite anonymous commenters still store their data with cloud providers, and would really prefer to believe that there are no worthwhile alternatives, hence all the ad hominem attacks.

Fortunately, each polite and humane comment makes up for a whole pile of bad ones. :)

Your GMail account CAN be hacked over insecure WiFi

Today The Next Web posted an episode of BBC Watchdog where it was demonstrated how a GMail account was hacked through insecure (WEP) WiFi.


For those of you still wondering, I’d like to confirm that it is indeed possible to hack a GMail account over insecure WiFi: GMail does indeed always send your password through secure HTTP (SSL) so that this can’t be directly hacked, BUT, by default, the rest of your session happens through normal clear-text HTTP.  The Watchdog episode of course gives absolutely no technical details, but it’s most probably the “sidejacking” attack first published by Robert Graham, where the attacker reads the cookies of the post-authentication HTTP traffic and uses them to fool GMail into thinking that they are in fact the legitimate owners attacked GMail account.  This attack works on other webmail and -service providers too.

In short, if you EVER use a network connection that you don’t trust, simply change the “http:” in your URL bar to “https:”, or, even better, change your browser connection to “Always use https” on the GMail Settings – General page.   With both of these solutions, the whole connection will use secure HTTPS (SSL), and cookies can’t be sidejacked.

The drawback of the secure setting is that your GMail access will be slightly slower than usual:  The encryption costs more compute time at both ends, and the transmission of data is slightly less efficient.