On Mon, Oct 04, 2021 at 08:11:25PM -0700, Matt Joras wrote: > I will hop in here one more time. It was not a botched BGP update. The DNS > disappearance was an unfortunate but preventable side effect of a global > backbone issue. Had DNS been functioning everything still would have been > down: https://engineering.fb.com/2021/10/04/networking-traffic/outage/ It was a botched BGP configuration issue, and when BGP advertisements were updated across the global backbone, caused large portions of Facebook's network to not be reachable. The fact that this brought down Facebook's DNS appears to have significantly increased its downtime, since Facebook's engineers couldn't authenticate to the servers needed to fix the problem. There are certainly a large number of operational questions which this brings up --- why didn't Facebook have their own internal backbone networks, with their own internal split-view DNS? Why didn't have ways so their SRE's could get direct access to some of the servers in their data centers which didn't depend on access via the public or internal Internet backone networks? These are however, out of scope of the IETF, because it has to do with how a particular site configures its networks. > On Mon, Oct 4, 2021, 7:44 PM Phillip Hallam-Baker <phill@xxxxxxxxxxxxxxx> > wrote: > > > Yes, DNS was involved but referring it to the DNS operations list > > presupposes that it was a DNS operations issue and not an architecture > > issue. The DNS query protocol is brittle. Maintaining the availability of > > core DNS is ruinously expensive because registries and large services have > > to survive DDoS attacks. Sure, Facebook flubbed a BGP update. But why did > > they need to go there in the first place? When a BGP flub brings down DNS, > > the most likely culprit is ANYCAST. Let's take a look at Facebook's nameservers: facebook.com. 172786 IN NS b.ns.facebook.com. facebook.com. 172786 IN NS d.ns.facebook.com. facebook.com. 172786 IN NS c.ns.facebook.com. facebook.com. 172786 IN NS a.ns.facebook.com. b.ns.facebook.com. 143822 IN A 129.134.31.12 b.ns.facebook.com. 163741 IN AAAA 2a03:2880:f0fd:c:face:b00c:0:35 d.ns.facebook.com. 164470 IN A 185.89.219.12 d.ns.facebook.com. 168652 IN AAAA 2a03:2880:f1fd:c:face:b00c:0:35 c.ns.facebook.com. 164249 IN A 185.89.218.12 c.ns.facebook.com. 167948 IN AAAA 2a03:2880:f1fc:c:face:b00c:0:35 a.ns.facebook.com. 143821 IN A 129.134.30.12 a.ns.facebook.com. 166558 IN AAAA 2a03:2880:f0fc:c:face:b00c:0:35 These are any ANYCAST addresses. Cheers, - Ted
