Re: Google threatens to break Gmail

[Ted Lemon <ted.lemon@xxxxxxxxxxx>]

On Oct 26, 2015, at 2:45 PM, John C Klensin <john-ietf@xxxxxxx> wrote:

If we decide that the long-established semantics are the right
ones, then I think our email standards deserve to die, because
they don't currently work.

Ted, I think millions of users, passing around tens or hundreds
of millions of messages around a day, would probably disagree
with "don't currently work" or at least dismiss it as rather
extreme hyperbole.

I will admit to using an extreme form of the term "work," which is "work without massive difficulties."   Most of the massive difficulties happen behind the scenes, so we fortunates do not have to deal with them, but the lengths mail providers like Google have to go to to keep spam out of our inboxes cast their shadow over the experience of every email user.   Web registration systems now tell users to whitelist mail from their domains, which would be cool if it could be done automatically rather than through manual intervention, and users are now accustomed to searching for important mail that hasn’t arrived in their junk folders.   I suspect the annual number of person-hours spent on these two tasks would humble us.

My experience of "badly broken": pre-filter, for my email address (mellon@xxxxxxxxx) alone, spam is currently arriving at a rate of about three messages per minute.   Why?   Because the design assumptions for email did not account for the fact that in the wild, email is an ecosystem, not a cooperative venture, and there is money to be made with scattershot spam, and money to be made filtering it.

I’ve attempted to implement a whitelist on my server to make sure that mail from people I already know will get through.   What I’ve discovered is that aside from the big providers, nobody sets their SPF up right, so I can’t rely on it to validate whitelisted senders: I either have to just hope for the best, and accept the occasional joe job attack, or else I whitelist IP addresses, or come up with heuristics like "if the MX for a domain points to google, pretend that the domain owner set up SPF according to Google’s docs."

The amount of brainpower that’s required to keep this rickety train on the rails is astonishing.   It is no longer the case that someone like you or I with the resources of an individual can have a reasonably painless experience of operating an SMTP server.   To my mind, this means that SMTP does not "work."   There is no dependable method by which I can ask the question "did this email message come from the source that it claims to have come from" and get an answer.

That’s what I mean by "works."   And that’s why I have every sympathy with mail providers who are throwing up their hands in disgust and saying I’m going to be draconian, even though the specs don’t technically allow it.   If we want something different to happen, we have to figure out a way to allow it to happen, and not just say there’s no problem and Google ought to follow the specs.

Perhaps I haven't been looking in the right places, but I
haven't heard Google claim that email is "badly broken", much
less "doesn't work".   What I have heard is some claims about
blocking of some messages originating from bogus or unauthorized
senders.  That is a sender authentication problem, not a "broken
email protocol" one.

It’s true that if we had a reliable way of validating senders, SMTP could continue to operate.   And it might even be that if we had this mechanism in place, spammers would stop dumping crap on my mail server, so I wouldn’t have to pay for gigabytes per day of useless traffic to my server.   But in practice, it would be much better to use a protocol that didn’t even trigger a data transfer until the sender had been validated.

Equally important, if Google really
cares about either sender authentication or verification that a
sender who uses a particular backward-pointing address today is
the same entity who used it yesterday, we know of a large
variety of ways to approximate at least the latter.  The
observation that Google isn't doing any of those things, even
the ones they could support with a very large fraction of their
users and without protocol changes, suggests that isn't the
issue.

Yes, we do know ways to do this.   Most of them are too expensive to be practical.

So, if you are going to claim that our existing standards don't
work, I think it would be good to have a clear explanation of
what you mean and what, precisely, doesn't work.  Of course, I
can only hope that, contrary to your apparent claim, this
message will reach you in spite of non-working protocols and you
will be able to reply.

I have all the IETF mailing lists whitelisted, and IETF has a correct SPF setup (and, actually, I get my IETF mail through nominum’s servers, and Nominum pays Google a lot of money to filter spam, and they use their big data analytics to do a good job of that, so in that sense it does indeed "work," but I shudder to think what the carbon footprint of each valid delivered message is if you amortize the cost over the total number of messages that had to be transmitted and examined.