Andy Heydon

Were the Mayans right? I hadn’t received any new email in Gmail since 2:09 on the afternoon of 12/12/12. Was this the beginning of the end?

I normally have a steady drip of messages into my personal inbox throughout the day, even more so in this holiday period as companies that I have a legitimate relationship with, advise me of their latest offers and free shipping deals. But there it was, on the evening of the 12th — absolute silence. The fact that I as expecting some important messages increased my tension.

Only after some manual refreshing did Gmail inform me that it was having problems retrieving my email, prompting me to dig deeper and eventually eliciting the rather terse message:

SSL Security Error.
Server returned error “SSL error: self signed certificate”

My personal email is hosted by a server that sits in the UK and is managed by my brother. I just use Gmail as a convenient client so that I am not tied to a particular desktop, the spam filtering is first rate without any manual training, and the messages are stored safely on a server. The error message would imply that our server was configured incorrectly and, given that early afternoon in Portland is late evening in the UK, a time when a brother would be making changes to a server, would imply that something had been messed up. I know he doesn’t use Gmail as a client, so he may not be aware of the problems he has caused. But could I raise his attention with texts? No I could not!

Well, Kevan, I hereby apologize for besmirching your good name in my thoughts. You were entirely blameless in this escapade. Through some digging, it appears that Google decided to change their procedures and enforce a strict SSL policy. They would now only connect to a server if it has a valid, signed SSL certificate. Any mail server using a self-signed certificate, which are a common occurrence amongst personally managed mail servers such as ours, would be refused.

As an aside, the error message “Server returned error”, is poorly written because it is not clear as to whose server we are talking about. It is not an error that our mail server is returning a self-signed certificate — that is a legitimate thing to do. The problem is that Google is not allowing such an activity. This is not an SSL error, it is a policy of not accepting certain kinds of configurations. The error message is just lazy engineer speak that fails to convey the correct issue.

Now I don’t disagree with the policy change as it helps to protect from man-in-the-middle attacks, but I do condemn the implementation of the change, and it demonstrates that Google is an engineering company and doesn’t understand customer service.

In any production system, if you are going to introduce a change that a) will disrupt the service, or b) force the customer to perform an action, or c) cause the customer to pay some money, then you need to proactively communicate that change. Google, with this SSL policy enforcement, hit that trifecta and absolutely should have told everyone of the change.

The solution to the problem is for us to purchase an SSL certificate from a reputable authority. No big deal, except that this takes time because our identity has to be verified, except that it costs money, and we have no access to email during the transition. I had to hurriedly configure a desktop email client that I could authorize to overlook a self-signed certificate, but this will be a temporary crutch until we can install a signed certificate, and is something that I shouldn’t have to do. From Google’s perspective that is tempting trouble because I might like the new system and give up on Gmail altogether. Clearly the policy change was not fully thought through.

It would have been trivial for Google to determine all the accounts that fetched email from a remote server and verified which of those servers had a self-signed certificate. It should have then sent those accounts an email with the details, reasons and implications of the upcoming change, along with a timeline for its implementation. In this particular case, because it requires the purchase of an SSL certificate, there should have been at least a week’s notice. You cannot just pull the plug on a service if the solution requires a significant time to implement. It shows a total lack of respect for your customers and their needs.