Yesterday there was a big Facebook outage caused by BGP. I’ve been vaguely interested in learning more about BGP for a long time, so I was reading a couple of articles.
I got frustrated because none of the articles showed me how I could actually look up information related to BGP on my computer, so I wrote a tweet asking for tools.
I got a bunch of useful replies as always, so this blog post shows some tools you can use to look up BGP information. There might be an above average number of things wrong in this post because I don’t understand BGP that well.
I can’t publish BGP routes
One of the reasons I’ve never learned much about BGP is – as far as I know, I don’t have access to publish BGP routes on the internet.
With most networking protocols, you can pretty trivially get access to implement the protocol yourself if you want. For example you can:
- issue your own TLS certificates
- write your own HTTP server
- write your own TCP implementation
- write your own authoritative DNS server for your domain (I’m trying to do that right now for a small project)
- set up your own certificate authority
But with BGP, I think that unless you own your own ASN, you can’t publish routes yourself! (you could implement BGP on your home network, but that feels a bit boring to me, when I experiment with things I like them to actually be on the real internet).
Anyway, even though I can’t experiment with it, I still think it’s super interesting because I love networking, so I’m going to show you some tools I found to learn about BGP :)
First let’s talk through some BGP terminology though. I’m going to go pretty fast because I’m more interested in the tools and there are a lot of high level explanations of BGP out there (like this cloudflare post).
What’s an AS (“autonomous system”)
The first thing we need to understand is an AS. Every AS:
- is owned by an organization (usually a large organization like your ISP, a government, a university, Facebook, etc)
- controls a specific set of IP addresses (for example my ISP’s AS includes 247,808 IP addresses)
- has a number (like 1403)
Here are some observations I made about ASes just by doing some experimentation:
- Some fairly big tech companies don’t have their own AS. For example, I looked up Patreon on BGPView, and as far as I can tell they don’t own as AS – their main site (patreon.com, 22.214.171.124) is in Cloudflare’s AS.
- An AS can include IPs in many countries. Facebook’s AS (AS32934) definitely has IP addresses in Singapore, Canada, Nigeria, Kenya, the US, and more countries.
- It seems like IP address can be in more than one AS. For example, if I look up 126.96.36.199, it has 2 ASNs associated with it – AS6130 and AS21581. Apparently when this happens the more specific route takes priority – so packets to that IP would get routed to AS21581.
what’s a BGP route?
There are a lot of routers on the internet. For example, my ISP has routers.
When I send my ISP a packet (for example by running
ping 188.8.131.52), my ISP’s routers
needs to figure out how to actually get my packet to the IP address
The way the router figures this out is that it has a route table – it has a list
of a bunch of IP ranges (like
184.108.40.206/23), and routes it knows about to
get to that subnet.
Here’s an example of a real route for
220.127.116.11/23: (one of Facebook’s subnets). This one isn’t from my ISP.
11670 32934 18.104.22.168 from 22.214.171.124 (126.96.36.199) Origin IGP, metric 0, valid, external Community: 3856:55000 Last update: Mon Oct 4 21:17:33 2021
I think that this is saying that one path to
188.8.131.52 is through the
184.108.40.206, which is on its local network. So the router might send
my ping packet to
220.127.116.11 next, and then
18.104.22.168 will know how to
get it to Facebook. The two numbers at the beginning (
11670 32934) are ASNs.
My understanding of BGP is very shaky, but it’s a protocol that companies use to advertise BGP routes.
What happened yesterday with Facebook is that they basically made BGP announcements withdrawing all their BGP routes, so every router in the world deleted all of its routes related to Facebook, so no traffic could get there.
Okay, now that we’ve covered some basic terminology, let’s talk about tools you can use to look at autonomous systems and BGP!
tool 1: look at your ISP’s AS with BGPView
To make this AS thing less abstract, let’s use a tool called BGPView to look at a real AS.
My ISP (EBOX) owns AS 1403. Here are the IP addresses my ISP owns. If I look up my
computer’s public IPv4 address, I can check that it’s one of the IP addresses
my ISP owns – it’s in the
BGPView also has this graph of how my ISP is connected to other ASes
traceroute -A and
Okay, so we’re interested in autonomous systems. Let’s see which ASes I go through from
mtr both have options to tell you the ASN for every IP you go through. The flags are
traceroute -A and
mtr -z, respectively.
Let’s see which autonomous systems I go through on my way to facebook.com with
$ mtr -z facebook.com 1. AS??? LEDE.lan 2. AS1403 104-163-190-1.qc.cable.ebox.net 3. AS??? 10.170.192.58 4. AS1403 0.et-5-2-0.er1.mtl7.yul.ebox.ca 5. AS1403 0.ae17.er2.mtl3.yul.ebox.ca 6. AS1403 0.ae0.er1.151fw.yyz.ebox.ca 7. AS??? facebook-a.ip4.torontointernetxchange.net 8. AS32934 po103.psw01.yyz1.tfbnw.net 9. AS32934 22.214.171.124 10. AS32934 edge-star-mini-shv-01-yyz1.facebook.com
This is interesting – it looks like we go directly from my ISP’s AS (1403) to Facebook’s AS (32934), with an “internet exchange” in between.
I’m not sure what an internet exchange is but I know that it’s an extremely important part of the internet. That’s going to be for another day though. My best guess is that it’s the part of the internet that enables “peering” – like an IX is a server room with a gigantic switch with infinite bandwith in it where a bunch of different companies put their computers so they can send each other packets.
mtr looks up ASNs with DNS
I got curious about how mtr looks up ASNs, so I used strace. I saw that it looked like it was using DNS, so I ran dnspeep, and voila!
$ sudo dnspeep ... TXT 126.96.36.199.origin.asn.cymru.com 192.168.1.1 TXT: 1403 | 188.8.131.52/20 | CA | arin | 2014-08-14, TXT: 1403 | 184.108.40.206/19 | CA | arin | 2014-08-14, TXT: 1403 | 220.127.116.11/17 | CA | arin | 2014-08-14 ...
So it looks like we can find the ASN for
18.104.22.168 by looking up the
txt record on
22.214.171.124.origin.asn.cymru.com, like this:
$ dig txt 126.96.36.199.origin.asn.cymru.com 188.8.131.52.origin.asn.cymru.com. 13911 IN TXT "1403 | 184.108.40.206/19 | CA | arin | 2014-08-14" 220.127.116.11.origin.asn.cymru.com. 13911 IN TXT "1403 | 18.104.22.168/17 | CA | arin | 2014-08-14" 22.214.171.124.origin.asn.cymru.com. 13911 IN TXT "1403 | 126.96.36.199/20 | CA | arin | 2014-08-14"
That’s cool! Let’s keep moving though.
tool 3: the packet clearing house looking glass
PCH (“packet clearing house”) is the organization that runs a lot of internet exchange points. A “looking glass” seems to be a generic term for a web form that lets you run network commands from another person’s computer. There are looking glasses that don’t support BGP, but I’m just interested in ones that show you information about BGP routes.
Here’s the PCH looking glass: https://www.pch.net/tools/looking_glass/.
In the web form on that site, I picked the Toronto IX (“TORIX”), since that’s what
mtr said I was using to go to facebook.com.
thing 1: “show ip bgp summary”
Here’s the output. I’ve redacted some of it:
IPv4 Unicast Summary: BGP router identifier 188.8.131.52, local AS number 3856 vrf-id 0 BGP table version 33061919 RIB entries 513241, using 90 MiB of memory Peers 147, using 3003 KiB of memory Peer groups 8, using 512 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd ... 184.108.40.206 4 1403 484672 466938 0 0 0 05w3d03h 50 ... 220.127.116.11 4 32934 482088 466714 0 0 0 01w6d07h 38 18.104.22.168 4 32934 482019 466475 0 0 0 01w0d06h 38 ... Total number of neighbors 147
My understanding of what this is saying is that the Toronto Internet Exchange (“TORIX”) is directly connected to both my ISP (EBOX, AS 1403) and Facebook (AS 32934).
thing 2: “show ip bgp 22.214.171.124”
Here’s the output of picking “show ip bgp” for
126.96.36.199 (one of Facebook’s IP addresses):
BGP routing table entry for 188.8.131.52/23 Paths: (4 available, best #4, table default) Advertised to non peer-group peers: 184.108.40.206 11670 32934 220.127.116.11 from 18.104.22.168 (22.214.171.124) Origin IGP, metric 0, valid, external Community: 3856:55000 Last update: Mon Oct 4 21:17:33 2021 11670 32934 126.96.36.199 from 188.8.131.52 (184.108.40.206) Origin IGP, metric 0, valid, external Community: 3856:55000 Last update: Mon Oct 4 21:17:31 2021 32934 220.127.116.11 from 18.104.22.168 (22.214.171.124) Origin IGP, metric 0, valid, external, multipath Community: 3856:55000 Last update: Mon Oct 4 21:17:27 2021 32934 126.96.36.199 from 188.8.131.52 (184.108.40.206) Origin IGP, metric 0, valid, external, multipath, best (Older Path) Community: 3856:55000 Last update: Mon Oct 4 21:17:27 2021
This seems to be saying that there are 4 routes to Facebook from that internet exchange.
the quebec internet exchange doesn’t seem to know anything about Facebook
I also tried the same thing from the Quebec internet exchange QIX (which is
presumably closer to me, since I live in Montreal and not Toronto). But the QIX
doesn’t seem to know anything about Facebook – when I put in
just says “% Network not in table”.
So I guess that’s why I was sent through the Toronto IX and not the Quebec one.
more BGP looking glasses
Here are some more websites with looking glasses that will give you similar
information from other points of view. They all seem to support the same
show ip bgp syntax, maybe because they’re running the same software? I’m not sure.
There seem to be a LOT of these looking glass services out there, way more than just those 3 lists.
Here’s an example session with one of the servers on this list: route-views.routeviews.org. This time I connected via telnet and not through a web form, but the output looks like it’s in the same format.
$ telnet route-views.routeviews.org route-views>show ip bgp 220.127.116.11 BGP routing table entry for 18.104.22.168/24, version 1053404087 Paths: (23 available, best #2, table default) Not advertised to any peer Refresh Epoch 1 3267 1299 32934 22.214.171.124 from 126.96.36.199 (188.8.131.52) Origin IGP, metric 0, localpref 100, valid, external path 7FE0C3340190 RPKI State valid rx pathid: 0, tx pathid: 0 Refresh Epoch 1 6939 32934 184.108.40.206 from 220.127.116.11 (18.104.22.168) Origin IGP, localpref 100, valid, external, best path 7FE135DB6500 RPKI State valid rx pathid: 0, tx pathid: 0x0 Refresh Epoch 1 701 174 32934 22.214.171.124 from 126.96.36.199 (188.8.131.52) Origin IGP, localpref 100, valid, external path 7FE1604D3AF0 RPKI State valid rx pathid: 0, tx pathid: 0 Refresh Epoch 1 20912 3257 1299 32934 184.108.40.206 from 220.127.116.11 (18.104.22.168) Origin IGP, localpref 100, valid, external Community: 3257:8095 3257:30622 3257:50001 3257:53900 3257:53904 20912:65004 path 7FE1195AF140 RPKI State valid rx pathid: 0, tx pathid: 0 Refresh Epoch 1 7660 2516 1299 32934 22.214.171.124 from 126.96.36.199 (188.8.131.52) Origin IGP, localpref 100, valid, external Community: 2516:1030 7660:9001 path 7FE0D195E7D0 RPKI State valid rx pathid: 0, tx pathid: 0
Here there are a few options for routes:
3267 1299 32934
701 174 32934
20912 3257 1299 32934
7660 2516 1299 32934
I think the reason there’s more than one AS in all of these is that
184.108.40.206 is a Facebook IP address in
Toronto, so this server (which is maybe on the US west coast, I’m not sure) is
not able to connect to it directly, it needs to go to another AS first. So all of the routes have one or more ASNs
The shortest one is 6939 (“Hurricane Electric”), which is a “global internet backbone”. They also have their own hurricane electric looking glass page.
tool 4: BGPlay
All the other tools so far have just shown us the current state of Facebook routing where everything is fine, but this 4th tool lets us see the history of this Facebook BGP internet disaster! It’s a GUI tool so I’m going to include a bunch of screenshots.
The tool is at https://stat.ripe.net/special/bgplay. I typed in the IP address 220.127.116.11 (one of Facebook’s IPs), if you want to play along.
First, let’s look at the state of things before everything went wrong. I clicked in the timeline at 13:11:28 on Oct. 4, and got this:
I originally found this very overwhelming. What’s happening? But then someone on Twitter pointed out that the next place to look is to click on the timeline right after the Facebook disaster happened (at 18:38 on Oct. 4).
It’s pretty clear that something is wrong in this picture – all the BGP routes are gone! oh no!
The text at the top shows the last Facebook BGP route disappearing:
Type: W > withdrawal Involving: 18.104.22.168/24 Short description: The route 50869, 25091, 32934 has been withdrawn. Date and time: 2021-10-04 16:02:33 Collected by: 20-22.214.171.124
If I then click the “fast forward” button, we see the BGP routes start to come back:
The first one announced is
137409 32934. I don’t think this is actually the
first one announced though – there are a lot of route announcements inside the
same second (at 2021-10-04 21:00:40), and I think the ordering inside BGPlay is arbitrary.
If I click the “fast forward” button again, more and more routes start to come back and routing starts to go back to normal
I found looking at this outage in BGPlay really fun, even though the interface is pretty confusing at first.
maybe it is important to understand a little about BGP?
I started out this post by saying you can’t change BGP routes BGP, but then I remembered that in 2016 or 2017 there was a Telia routing issue that caused us some minor network at work. And when that happens, it is actually useful to understand why your customers can’t reach your site, even if it’s totally out of your control. I didn’t know about any of these tools at that time but I would have liked to!
I think for most companies all you can do to respond to outages caused by someone else’s bad BGP routes is “do nothing and wait for it to get fixed”, but it’s nice to be able to confidently do nothing.
some ways to publish BGP routes
If you want to (as a hobbyist) actually publish BGP routes, here are some links from the comments:
- a guide to getting your own ASN
- dn42 seems to have a playground for BGP (it’s not on the public internet, but it does have other people on it which seems more fun than just doing BGP by yourself at home)
that’s all for now
I think there are a lot more BGP tools (like PCH has a bunch of daily snapshots of routing data which look like fun), but this post is already pretty long and there are other things I need to do today.
I was surprised by how much information I could get about BGP just as a regular person, I always think of it as a “secret network wizard” thing but apparently there are all kind of public machines anybody can just telnet to and use to look at the route tables! Who knew!