Weekly Head Voices #111: A swift hack.

Well hello friends! In this here,the one hundred and eleventh edition of the Weekly Head Voices, I present a personal view of selected events that took place in the time between Monday, July 25 and Sunday, August 14 of 2016.

Post summary: HackerNews FastMail to Gmail retrospective (WARNING NERD CONTENT), Craft Beer tips, Swift Playgrounds (teach your kids to code!) and a tiny bit of backyard philosophy at the end.

The HackerNews effect

When I submitted my Moving 12 years of email from GMail to Fastmail blog post to HackerNews last Monday, it was after some consideration, and with considerable trepidation. The previous time a similar post of mine was picked up by the HN frontpage (and reddit) in 2013, it resulted in a great number of fairly harsh insults flung my way (the harsh ones seem to hit much harder than the many more constructive ones can compensate for). The problem is not having your work criticised, it’s rather being ad hominem’ed into the ground.

The internet can unfortunately get quite bad that way.

However, this time the internet was in a good mood!

As a Z-list (aka hobbyist) blogger, I already get quite excited when even two people find something entertaining or educational in my blog, so you can imagine my excitement when I saw my blog stats jump into the thousands during the first hour after hitting the HN front page.

646 upvotes, 365 comments (in the HN discussion) and more than 50 thousand blog post views later (!!), my asbestos suit is still in storage, and the box of kleenex (for drying up my tears) is still unopened. I am very happy with the healthy and mostly happy discussion both on HN and here on the blog.

A taste of my secret beer notes

In my travels around the world (ok, maybe just in a few hundred kilometre radius of where I live) I taste many exotic and strange drinks (okay, maybe just the local craft beers). Here’s an excerpt from my top secret beer notes for your reading and hopefully soon tasting pleasure:

Stellies Bosch Weiss
Stellies Bosch Weiss

The Stellies Bosch Weiss is is a refreshing white beer which I can imagine enjoying much more in 35 celsius. However, I find it way too gassy for my taste and thus give it 0.4 on the Celis White scale. As everyone knows, Celis White is the best white beer in the world. I’m also not the biggest fan of CBC’s Krystal Weiss because of its gassiness. This could also be a German vs Belgian thing.

The Darling Brew Bone Crusher is probably a 0.8 on the Celis scale, and scores mega bonus points for choosing such a heavy metal name.

Citizen Beer's Patriot Lager
Citizen Beer’s Patriot Lager

Citizen Beer has a real knack for naming beers; so this weekend I was able to do my patriotic duty (ha ha) by enjoying their Patriot lager. Fortunately, they also have a great knack for making lovely beers. This lager is more than hoppy enough to remind you that it’s a craft, but at the same time it is refreshingly light.

Devil's Peak Lager
Devil’s Peak Lager

Not completely coincidentally, I also tasted the Devil’s Peak Lager this weekend. There are subtle differences between it and the Patriot which I will only be able to describe after more tasting. For now: Same lightness, but with enough hop. Highly enjoyable.

Devil's Peak Pale Ale
Devil’s Peak Pale Ale

As I’ve mentioned before, it’s no coincidence that happiness and hoppiness look so similar. The Devil’s Peak Pale Ale was a superbly hoppy (about 239% more hoppy than the lager) and full flavoured conclusion to the weekend. In spite of its full body this Cape Town beer’s alcohol content is lower than I would have expected: Only 4%.

First impressions of Swift Playgrounds

I might or might not have acquired a new iPad Air 2 with the primary motivation of being able to test the new Swift Playgrounds on Genetic Offspring Unit #1 (now 10 years old). This is a new and attractive iPad-only app that has been designed by Apple to teach kids how to program in Swift. Note that you’ll have to upgrade to iOS 10 Beta to get the app, at least until iOS 10 is officially released. One of the exercises looks like this (image taken from the website):

Screen Shot 2016-08-14 at 6.25.43 PM

Previous experiments with scratch and with processing have met with limited success but no permanence. However, GOU#1 is an iPad fanatic, and the potential of getting her addicted to the programming bug is just too fantastic to let this opportunity go by.

On the first day, she was already writing functions and for loops in Swift in order to navigate a cute alien solving puzzles on a 3D landscape.  I was looking over her shoulder now and then: The educational content and execution of the app is impressive. Besides the built-in puzzle worlds such as the 3D one pictures above, you can create your own Swift projects from scratch. These projects can use iPad hardware such as bluetooth and the camera, but we’re not quite there yet. Apple has also promised to keep on expanding the educational content.

I’m really crossing my fingers that GOU#1 will keep at it. If Swift Playgrounds helps to get her programming, I might just have to go full fanboy.

∞♥

This past Friday at the breakfast table, Genetic Offspring Unit #2 asked her mom how much she thought GOU#2 loved the baby GOU#3, upon which the mom asked “How much do you love GOU#3?”.

GOU#2 answered: Infinitely much!

My fragile parental unit heart almost exploded with happiness at this point. It’s what I tell GOU#1 and #2 when I put them to bed, and every morning when I drop them at school. (GOU#3 herself does not yet parse our language.)

Ok kids, it looks like there’s an infinite amount of the good stuff to go around. You know what to do!

bettys_bay_somewhere
I went jogging (or rather walking with a two-step now and then). Enjoyment of surroundings and physical activity was quite intense at this point in spacetime: 15:55 on August 7, 2016; GPS coordinates in EXIF data.

Moving 12 years of email from GMail to FastMail

In 2013, when it became clear, primarily through Edward Snowden’s heroic actions, that the level of snooping by the US and other governments was far greater than any of us would have thought, I moved all of my data out of the US and of course blogged about it (that blog post has been read almost 70000 times; I think for many people this is an important issue).

This included migrating 60000 emails away from my beloved GMail (I got my GMail invite from The Vogon Poet on August 24, 2004. At that time, you could only get GMail by invitation. It was pretty exciting! (I have emails from before 2004, back to ’93 or ’94, but those are in a backup archive somewhere.)) to the little Synology DS213j standing next to my desk at the time.  This was all well and good behind the stable Dutch 100 Mbit/s down / 10 Mbit/s up cable connection I had, but when we decided to move back to South Africa, where home internet is a few years behind The Netherlands, I ended up having to pay for a virtual private server in Cape Town (to keep latency between me and my mail server manageable) and having to admin my own dovecot IMAP and postfix SMTP server.

Initially this was workable, until the Nth time that I had to interrupt my real job (which has nothing to do with mail servers) to apply a security patch or get the VPS booting again after a botched kernel upgrade. Besides that, I had to deal with keeping my server out of over-enthusiastic spam blacklists and whatnot. Also, inspite of mu4e, I did end up missing the fast graphical GMail web interface.

So, it was with a great deal of tail between my legs that on June 10, 2015 (I have a lab journal, remember) I went right back back to GMail. My mail setup, although pleasingly decentralised, was costing me too much time and hence actual money.

Fast forward to July 15, 2016 (there’s that lab journal again…) when, after receiving an email from Google asking me to indicate how exactly I would like them to use my data to customise adverts around the web, and after thinking for a bit about what kind of machine learning tricks I would be able to pull on you with 12 years of your email, I decided that I really had to make alternative plans for my little email empire.

Somehow FastMail came up and in one of those impulsive LET’S WASTE SOME TIME manoeuvres, I pressed the big red MIGRATE button!

The rest of this post is my mini-review of the FastMail service after almost 3 weeks of intensive use.

Importing mail from GMail

The main import & export window
The main import & export window
IMAP migration configuration dialog
IMAP migration configuration dialog

The Settings | Import & Export option in FastMail was easy to setup. It knows how to authenticate with GMail, even when you make use of two-factor authentication, like I do and you probably should.

The import takes place via the GMail IMAP interface. It’s important to remember that via the IMAP client, an email tagged in GMail with both important and info will appear in two different folders. Because of this, I did check the no duplicates checkbox, but still I noticed that my 15 GB FastMail evaluation mailbox was filling up more quickly than I would have expected.

After a support request which was responded to within minutes (bonus), I discovered the Quota Usage screen and could see that the duplicate detection did indeed not seem to work correctly during the import. Based on more tips from the support tech, I made use of the Mass delete or remove duplicates module (Settings | Folders | Scroll all the way to the bottom of the page) to delete thousands of duplicate emails during the import. This was indeed because of emails appearing in multiple IMAP folders due to their GMail tags. Note: Friend and reader stefanvdwalt reported the exact same mail duplication during import issue which in his case did go over quota, so do keep an eye on this!

After a day or so (during which I could email more or less normally) I received an import report from FastMail claiming that the import had been successful, except for this error:

Log: Fri Jul 15 17:49:17 2016; cpbotha/imap.gmail.com; Migrating folder Inbox -> Inbox
Log: Fri Jul 15 17:49:17 2016; cpbotha/imap.gmail.com; Creating local folder Inbox
Log: Fri Jul 15 17:49:17 2016; cpbotha/imap.gmail.com; Error migrating remote folder Inbox: Failed to create Inbox. IMAP Command : 'create' failed. Response was : no - Mailbox already exists

The import had managed to figure out that GMail Sent should map to FastMail Sent for example, but Inbox was probably too special to map in the same way. I fixed this by firing up my trusty Thunderbird, and using IMAP to drag and drop emails from my GMail Inbox to my shiny new FastMail Inbox.

In retrospect, I should have selected Create under new sub-folder in the IMAP migration configuration instead of Merge into existing folders. I discovered later that moving thousands of emails to a different folder is near instantaneous in the FastMail web-app.

What I like

Webmail speed

I live more or less at the southern tip of the African continent. My lowest latency connection with the rest of the internet is via undersea optic cable to Europe (about 140ms ping).

The FastMail web servers are in the USA, which is, as the ping flies, much further away. I was not expecting much from the webmail, but colour me surprised when I discovered that this felt subjectively faster than GMail (who have servers everywhere, even down here). Things remained snappy, even with all 50000 of my conversations imported.

As far as I can figure out, it seems that much of this is due to FastMail’s self-designed but open source IMAP-replacement called JMAP. JMAP has been designed for low latency, and for improved battery life. What it does differently, is batch requests together, and it also has optimisations specifically for interactive webmail.

The web-app has full support for keyboard shortcuts, which increases the subjective perception of speed.

Webmail search

For my purposes, search in FastMail is on par with that of GMail. I can dig up any of my emails, back up to 2004, in seconds.

FastMail advanced search interface
FastMail advanced search interface

What’s also very useful, is that you can turn any search into a virtual folder.

Tech support

This is one area where Google really can’t hold a candle to FastMail. If something goes wrong with your gmail account (this hardly ever happens, but it’s possible) it’s almost impossible to get hold of any kind of official tech support. Here’s a recent story where a GMail user’s account was summarily terminated. There was probably some kind of ToS infringment, but the user has no idea what or why, and has lost all access to their emails and contacts database.

So far I’ve contacted FastMail tech support twice: Once during my email migration, and once to confirm the absence of the “quote selected text in reply” feature (discussed below). In both cases, I was helped by real humans who responded very quickly and courteously to my support requests.

Email and contacts (and calendar) out of Google’s view

I’m still of the opinion that Google makes fantastic and valuable products. However, with all of their data mining know-how and resources, one has to decide how much of one’s personal information one is willing to trade in for the use of these fantastic products.

With FastMail, I have been able to extricate my significant email archive (2004 to 2016, 50000 conversations) as well as my contacts database. I’m still making use of Google Calendar, because of bunches of sharing going on with family members, but I have the option of moving that out also.

By the way, the FastMail Calendar web interface is more than capable (and pretty enough) to replace Google’s version.

What I don’t like

Missing integrations: Todoist

GMail, being as popular as it is, has tonnes of integrations with other apps. In my case, I will really miss the Todoist for Gmail extension. With this, I had a mini-todoist window inside my GMail, and I could turn any email into a task at the click of a button (or the press of a shortcut).

Because FastMail email URLs seem to be persistent, I use the Todoist Chrome extension’s “Add to Todoist” context menu action to add the URL and email subject as a task. This not as nice as the gmail-specific extension (the task goes immediately into the todoist inbox, without the possibility to edit metadata such as due date and tags).

Missing feature: Quote selection in reply

In Gmail and in Thunderbird, if you select text in an incoming email and then reply, that selected text is quoted in the reply email. Unfortunately, this feature is not available in the FastMail web-app, and they have no plans to implement it.

I use both the FastMail web-interface as well as Thunderbird, because of its great PGP email encryption and signature support (hey, find me on keybase, send me encrypted email!), so this issue is somewhat ameliorated. Still, it would have been nice.

Android app lag

I do have FastMail’s Android app on my telephone. The app is a Cordova / PhoneGap / CrossWalk style unit with real-time email push and notification via Google Cloud Messaging (this is a relatively energy-efficient way for android phones to get push notification and is natively supported by FastMail).

However, there is a few second lag when I open the inbox, so I prefer using the pro version of AquaMail, a great Android IMAP mail client. I have this set to 15 minute polling for new email, as IMAP IDLE (push, in other words) is not as battery efficient as GCM or Apple’s email push. Opening any folder or email in AquaMail is of course instantaneous, as the emails live on the phone.

That being said, I use the FastMail app for searching, which is just as fast and as effective as the web-app.

THAT being said, FastMail really needs to implement some sort of caching in the Android app for lightning fast folder and email access. (The FastMail app is quite attractive, I would prefer using it more.)

FastMail Android app Calendar screen, from the Google Play page.
FastMail Android app Calendar screen, from the Google Play page.

Niggle: Creating an email alias / incoming route automatically creates a new sending identity

FastMail can manage the DNS for any of the custom domains that you assign to it, which is super useful if you don’t already have a DNS service.

I already make use of webfaction’s DNS for all of my domains, so I chose to add DNS records to designate fastmail as the official MX for those domains. (All of this is explained clearly in the FastMail help.)

When you do this, you have to create an email alias for each incoming address you would like to receive mail for (you can also create a catchall, but this could result in more spam arriving in your inbox). For each and every alias, FastMail automatically creates an outgoing (from address) identity. While this is usually quite convenient, I have quite a number of incoming addresses, but I only ever send from a subset of these addresses, so the drop-down list with sending identities became quite unwieldy.

I deleted all of the unnecessary identities. What would help, would be if FastMail were to implement most-used-at-the-top sorting for that drop-down.

Other noteworthy points

Domain setup

For my most important domains, I have set FastMail to be the MX. I have also performed the necessary SPF and DKIM setup: FastMail gives super useful feedback in its configuration screens to help you with this. For these domains, I send mail directly via the FastMail SMTP servers, and mail is delivered directly to FastMail servers. Nice and simple.

Domain setup feedback screen.
Domain setup feedback screen.

For some other email accounts I have with clients, FastMail supports POP fetch from and SMTP send via foreign servers.

iOS Push support

If you use any Apple iOS devices to read your mail, you’ll be pleased to know that FastMail, with help from the big A, fully supports iOS push. This means battery efficient real-time incoming emails to make it even more difficult for you to focus on That One Really Important Thing.

Android contact syncing with CardDAV

With google contacts, syncing on Android just works, and it works really well. To sync my contacts with FastMail’s Address Book instead, I bought the pro version of the CardDAV android app for 24 South African Ront (that’s about EUR 1.5). This works as a sync provider, so once setup, the process is also pretty much transparent.

Final thoughts

So there you have it: A hopefully helpful story, with included mini-review, about my move from GMail to the FastMail service.

So far, my conclusion is that this is a service that is technically more than capable of replacing GMail, even for power users. Furthermore, FastMail’s primary (and in fact only) business model is to charge you money for making sure that you can keep on emailing like a boss. Together, this makes for an offer that I could not refuse.

P.S. Let me know in the comments if you would like me to add anything else to this post.

P.P.S. You can also join the lively Hacker News discussion of this post!

Weekly Head Voices #109: GABA

  • From now on, I would like to limit WHVs to bullets (really) or to named sections, to ease reading. DOWN WITH WALLS OF TEXT!
  • After a multi-year, completely coincidental, break from medical imaging, I am back to The Real Business since the start of July. I am super excited about the plans we’re cooking up and executing. I can obviously not say too much, unless beer is involved, or you hang around here for muuuuuch longer. I think I am allowed to mention digital pathology and machine learning and beer.
  • Last week we road-tripped up the East Coast to St Francis Bay, via Oudtshoorn and the Cango Caves.
    • Pro tip: When road tripping with more than 0 (zero) children (babies count double; sick babies +5 hit points), and you have to stay overnight somewhere, invest extra in the biggest suite you (or your children’s college fund) can afford.
    • On the beach in St Francis Bay (right in the middle of winter, you still seem to get these lovely balmy beach days), it seemed that everybody was surfing. Whole families, with the mom, the dad, all the kids, and grandma and grandpa, were all on various sizes of surfboards in the sea catching some waves.
    • Here’s a photo from the furthest point on what I call “Not The Ugliest Jogging Route in The World” (in St Francis Bay):

20160714_170901

  • Last night I accidentally discovered that I can pinpoint the exact weekend and location when and where I first tasted my favourite trappist beer (EVER), namely Rochefort #8. It’s all written up in this 2003 post.
  • When Google send me an email this weekend asking me exactly how I would like them to use my email (yes, a few months ago I migrated my mail empire back to Google because my self-hosting experiment had started to cost me time and money) to show me custom advertisements, I was reminded that I do actually find the machine learning models they’re building about me quite creepy, and that perhaps I would prefer not also handing them 12 years of emails to make their models more accurate. There and then I migrated said 12 years of emails from GMail to FastMail. So far I’m really impressed by the product, mostly due to the speed and the user experience of the web-app. There might be a more detailed post in the near future, let me know if you’re interested.
  • Most surprising and interesting (to me) new scientific discovery of the week: A team of scientists at Northeastern University in Boston have shown that one of the many kinds of bacteria living in your stomach eats exclusively GABA, a really important brain chemical (neurotransmitter) that plays a role in keeping you calm. Based on this and other work, it looks like the bacteria in your tummy, also known as your gastrointestinal microbiota, besides being crucially important to your digestive system and your general survival, probably also play quite an important role in your psyche. I find this slightly mind-blowing.

Have a great week kids, I hope to see you on the other side.

P.S. Those bullets made for quite an impressive wall of text, didn’t they?