Monitoring BGP Feeds

Two weeks! I can't believe it has been that long since my last post. That's just crazy!

I know that I can't make it up to you but I will share something I just finished up.

BGP is cool. It is so cool to turn it up on a new router and see the entire internet's routing table with a simple command "sh ip bgp". What's cooler than that?

Feeding that table into a database, that's what!

Putting your BGP feed into a database enables all sorts of cool things. As a matter of fact, some of these are so cool I haven't even thought of them yet! For the last hour or so I've been busy working on getting this going. Here's what you will need:

- BGP feed from an upstream provider, connected to a router
- Linux machine running Quagga
- Linux machine with Perl installed (can be same machine as Quagga, mine is)

First you will need to configure your router with BGP enabled:

en

conf t

ip as-path access-list 1 deny .*

router bgp [your ASN]
neighbor [Quagga IP] remote-as 64512
neighbor [Quagga IP] transport connection-mode passive
neighbor [Quagga IP] description Quagga peer
neighbor [Quagga IP] filter-list 1 in


You will want to make sure that this machine is directly connected. If it isn't you need multihop BGP (which I won't cover right now). Here's what we're doing:

- The as-path with deny updates from your Quagga machine. We don't want some misconfiguration to actually affect your network. We just want some routes from the Cisco!

- Create the neighbor with remote-as 64512 (private ASN)

- Don't initiate a connection to this peer, let them connect to us (passive)

- Apply the filter-list to inbound traffic for this neighbor (don't allow updates from Quagga)

Now we need to configure Quagga. First, you will need to install it. Your distribution should have some packages for you. Use yum, apt, etc to grab it.

It will probably install some config files in /etc/quagga. We only want to setup bgp. This should be a good sample bgpd.conf to get you started:

hostname [your hostname]
password changeme
enable password changeme
log stdout
log syslog
service advanced-vty
!
router bgp 64512
bgp router-id [Quagga IP]
neighbor [Cisco IP] remote-as [your real ASN]
neighbor [Cisco IP] description Internet BGP Feed
neighbor 127.0.0.1 remote-as 64512
neighbor 127.0.0.1 description local db hookup
neighbor 127.0.0.1 port 9179
neighbor 127.0.0.1 passive
neighbor 127.0.0.1 filter-list 1 in
neighbor 127.0.0.1 next-hop-self
neighbor [some public ip] remote-as 64513
neighbor [some public ip] description Remote devel
neighbor [some public ip] passive
neighbor [some public ip] ebgp-multihop 255
neighbor [some public ip] filter-list 1 in
neighbor [some public ip] next-hop-self
!
access-list 1 permit [local class C network] 0.0.0.255
access-list 1 permit 127.0.0.1
access-list 1 deny any
access-list 10 permit [local class C network] 0.0.0.255
access-list 10 permit [remote ip]
access-list 10 deny any
!
ip as-path access-list 1 deny .*
!
line vty
access-class 1


After you apply this, you will want to start bgpd: "bgpd -n". It will tell you which vty you can connect to with telnet:

telnet localhost 2605

That should work. At this point, you should have a connection up to your main router:

sh ip bgp sum
BGP router identifier (deleted), local AS number 64512
29 BGP AS-PATH entries
0 BGP community entries

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
(deleted) 4 15092 580 129 0 0 0 02:06:29 9845
127.0.0.1 4 64512 167 944 0 0 0 00:58:59 0
(deleted) 4 64513 6 616 0 0 0 00:02:56 0

Total number of neighbors 3

Sweet! Hey, where did that connection from 127.0.0.1 come from? That looks strange...

That connection is the whole point of this exercise. Star2Star is interested in quality, internet routing and the relationship between the two (at least I am). Long ago we realized that having a full BGP feed and being able to analyze updates in real time could be a huge asset to our management and monitoring portfolio.

How could we do this? Within the last week it came up on NANOG. A guy named Bill Nash wrote a Perl script a while back using Net::BGP to connect to a BGP peer and dump it's routes into a MySQL database. This made much more sense than trying to pull them via SNMP or some other hackish mechanism.

Bill wrote this for another employer and no longer had access to it. Evidently enough people asked him about it off list for him to consider re-writing it. I wanted him to do more than consider... I wanted that script!

I contacted Bill and offered to support his work any way possible. This included setting up a read-only BGP feed for him. I didn't want to give him (no offense Bill) a direct connection to our Cisco router (crashing that would be BAD) so I came up with the setup above. The Quagga is read only (by configuration and with -n on the command line not even the local kernel can be updated). And the Cisco is read-only to the Quagga peer. Seems safe enough.

The Quagaa instance is merely a distributor for our BGP feed. That way I can mess with it all I want without any fear (or very little fear) of causing any problems for our main router. I can hammer it all I want with some alpha-quality perl scripts. Worst case (hopefully) I'll just hose Quagga if something goes wrong...

While waiting for Bill to get his Perl script going, I Googled BGP Perl to see if there was anything else out there. Sure enough, there is:

http://briangannon.com/2007/04/23/bgp-perl-route-analyzer/

This is a crude version of what I am looking for. I made a few minor changes because I needed it to run on the same machine as Quagga (already using TCP port 179). I also didn't want to have to run the perl script as root. Here is a mini-diff:

line 20:
-my $bgp = new Net::BGP::Process();
+my $bgp = new Net::BGP::Process( Port => 9179 );

Now when you follow the directions on Brian's blog to INSERT the BGP peer into the SQL table, make sure to just use localhost. Then the perl script will use port 9179 for itself. After all, if nothing needs to connect to it, who cares what the local port is (as long as the peer has been configured properly)? Quagga knows that peer 127.0.0.1 is on port 9179, and it works. Check this out:

6147167 | 4 | 216.134.176.0/22 | 2007-08-28 19:39:51 | 2 | Next Hop Changed,Metric Changed |
| 6147168 | 4 | 216.134.180.0/22 | 2007-08-28 19:39:51 | 2 | Next Hop Changed,Metric Changed |
| 6147169 | 4 | 216.85.83.0/24 | 2007-08-28 19:39:51 | 0 | Removal of network |
| 6147170 | 4 | 216.85.83.0/24 | 2007-08-28 19:39:57 | 1 | Added 216.85.83.0/24 |
| 6147171 | 4 | 216.85.83.0/24 | 2007-08-28 19:40:35 | 0 | Removal of network |
| 6147172 | 4 | 216.85.83.0/24 | 2007-08-28 19:41:22 | 1 | Added 216.85.83.0/24 |
| 6147173 | 4 | 207.250.244.0/23 | 2007-08-28 19:48:32 | 1 | Added 207.250.244.0/23

or the routes:

mysql> select * from route limit 9842,100;
+---------+-----------+------------------+-----------+--------+------------+---------------+
| id | router_id | prefix | next_hop | metric | local_pref | as_path |
+---------+-----------+------------------+-----------+--------+------------+---------------+
| 3045536 | 4 | 216.85.190.0/24 | 127.0.0.1 | | 100 | 15092 4323 |
| 3045638 | 4 | 198.102.2.0/24 | 127.0.0.1 | | 100 | 15092 4323 |
| 3045655 | 4 | 195.85.117.0/24 | 127.0.0.1 | | 100 | 15092 174 209 |
| 3045665 | 4 | 216.85.83.0/24 | 127.0.0.1 | 99999 | 100 | 15092 4323 |
| 3045666 | 4 | 207.250.244.0/23 | 127.0.0.1 | 99999 | 100 | 15092 4323 |
+---------+-----------+------------------+-----------+--------+------------+---------------+


That's from my MySQL db. Pretty cool, huh?

I'll be working more on this in the upcoming weeks. Bill will also be working on a much improved version of the BGP Perl script that I am working with now. I'll make sure to let everyone know how it goes!

Getting Multihomed - Part 3/3

Following up to one of my first posts. We FINALLY brought up BGP with all of our providers. A call from our CEO to some people at Verizon got some things moving again. I had the circuit up with BGP the same day. Pretty amazing, huh?

Anyways, now my problem was dealing with the limited memory and tcam allocation for unicast routes. If you recall, I ordered three full BGP feeds from three different providers. With the internet pushing 226,000 routes my 3750G wasn't going to cut it:

sh platform tcam utilization

CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values

Unicast mac addresses: 400/3200 13/44
IPv4 IGMP groups + multicast routes: 144/1152 6/26
IPv4 unicast directly-connected routes: 400/3200 13/44
IPv4 unicast indirectly-connected routes: 1040/8320 1023/8134
IPv4 policy based routing aces: 512/512 2/2
IPv4 qos aces: 512/512 8/8
IPv4 security aces: 1024/1024 23/23

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

So now I've got full feeds from three providers coming in. Luckily I read up on IOS route-map statements before I brought these BGP sessions up. Otherwise things could've gotten ugly. Here's what I started with:

ip as-path access-list 50 permit ^174$
ip as-path access-list 50 permit ^4323$
ip as-path access-list 50 permit ^701$

I started with just getting the ASNs we were directly connected to. And my tcam started to fill, but it wasn't close. I thought, hey, why not get some more routes while I can? I started to read up a bit more on route-maps and I figured out how to get other ASNs into my route table. I only want the networks of providers connected to my providers. Does that make sense?

Without being able to see the full table I would have no idea of what I was doing. What if I wanted Level(3)'s routes, for instance? I needed to see what was going on. Luckily an old client of mine runs FixedOrbit - the coolest site to look at BPG information. All I had to do was query my directly connected ASNs and start picking other routes I wanted. BGP would take care of the rest.

Here is a shortened version of what I ended up with:

ip as-path access-list 50 permit ^174$
ip as-path access-list 50 permit ^174_3356$
ip as-path access-list 50 permit ^174_33363$
ip as-path access-list 50 permit ^4323$
ip as-path access-list 50 permit ^4323_1668$
ip as-path access-list 50 permit ^4323_6983$
ip as-path access-list 50 permit ^4323_11456$
ip as-path access-list 50 permit ^701$
ip as-path access-list 50 permit ^701_19262$
ip as-path access-list 50 permit ^701_3356$

Now I have entries in my route table for my directly connected ASNs (174, 701, 4323) and some ASNs they are peered with - 3356, 33363, 1668, 6983, 11456, 19262. I don't have much room in my tcam but hey, that's what VXRs are for! Wow, I really want one of those (with an NPE-G2, of course) ;).

Getting Multihomed - Parts 1,2

WARNING: This is going to be a long post. You probably won't make it to the end. I guess that's what happens when I go this long without having a blog.

Since moving into our colo in Tampa way back when, Star2Star has been getting blended bandwidth from our colo facility (E-Solutions). First they had three providers (Verizon/MCI/UUnet, Global Crossing, Level3). Then two (Global Crossing, Level3).

Starting in February, Global Crossing started having some big problems. Mostly packet loss in a router in Miami. Not only did it happen frequently, it was bad (%50 - %60 loss). You can imagine what that does for VoIP...

We are obsessed with quality, so about four months ago we decided to get multihomed. Seems easy enough, right? Get the right equipment, order some circuits, do the BGP thing.

Let's start with the good equipment. We have been using the awesome Cisco Catalyst 3750 to form our redundant switch stack (two 3750G-24-TS-1U configured with STP to the colo's 4500). My buddy Anton Kapela at Five9s Data suggested them. How I love these switches:

- Good stacking (Cisco StackWise)
- Good performance (65.7mpps - that's over 65 million packets per second across the backplane)
- Good performance, with features. That's right, you can do QoS, ACLs, etc at wire speed, per port (within the limits of the TCAM, obviously).
- 24 +4 port density in 1U (24 GigE copper + four GBIC slots)
- More router-type functionality (with EMI software image - gives BGP, etc)

So with a little configuration I should be able to use one of these (right now I just grabbed a spare) to form our BGP capable router to aggregate all of these circuits. Remember those great services I talked about before? Remember the tcam? Turns out that it can only handle about 8,000 unicast routes before it starts to drop into software forwarding/otherwise start to act up. Not that big of a surprise, with the current full BGP table on the internet pushing 225K+ routes the 128MB of RAM in the 3750 wouldn't have done much good anyways. With our configuration (providers directly connected, aggregated routes only) 8000 unicast routes should be just fine. Sure we lose some end to end visibility, but it's still better than what we've got now.

It might not be the perfect equipment (it's no VXR, that's for sure) but it should get us started. Now I have to order some circuits...

We take a look at the customers we have now, the providers they have, the big providers in the area, all of that good stuff. We determine we would like to get (in no order):

- Cogent
- Verizon
- Time Warner

Cogent! Yes, I know, Cogent. Cogent sticks out on that list. Let's start with them.

Dealing with the sales guy was great. Very responsive. The price (I'm sure you know) was tough to beat. Even better than price, there was another perk...

Remember that huge global internet routing table I was talking about? There are many advertised networks in it, all with varying sizes. Some are a full Class A (/8), some are less than a Class C (/24). Or are they? It turns out that most providers/network admins/BGP snobs filter any announcement smaller than a full Class C (/24). Make sense. That table is out of control! Router memory is expensive! There is old equipment! What are those less-than-a-full-class-C small fries doing messing with BGP anyways?

What is someone with a currently small network supposed to do if they want to multihome? We need BGP to control our own routing and peer with other networks and providers. We don't have enough machines to justify the current ARIN/ISP policy of %25/%50 utilization for IP addresses to get a full class C.

It turns out that ARIN has been thinking of us. That's why there is ARIN policy 2001-2. This policy, in short, says that if you can prove you are multihoming, your ISP can give you a full Class C no questions asked. Out of the three providers mentioned above Cogent was the only provider that had even heard of this and they were more than happy to do the allocation. Thanks Cogent! (Why does everyone hate them so much?)

Don't get me wrong. I am really interested in IPv6. I know global IPv4 address space is shrinking. Hacks like NAT are running out too and it is only a matter of time before we run out of IP address space and the internet comes to a halt. Whatever. At the rate Star2Star is growing we'll need all of those IPs soon enough. When the internet really needs IPv6 all of the really smart internet god types will figure it out. I'm not worried.

So we have one provider. We have a class C. Now we need to get an ASN. Before we do that, we need to make the various contacts that ARIN requires. I started with applications for the OrgID and NocID (I think - can't remember exactly now). Much to my surprise, the turnaround time on both of these was less than one hour even though I didn't start until about 5:30 PM on a Friday. I guess they don't believe in P.O.E.T.S Day over at ARIN!

We get approved for the ASN in a day or so. Fax in the contract. Give them their $500. Now we need to wait until the good 'ol US Postal Service delivers TWO copies of our signed contract. I guess ARIN really wants to hold us to that one. Don't worry ARIN I promise we'll keep up our end of the deal. You've been great so far. Two days later (USPS Florida -> Virginia) our ASN (15092) is approved and will be in WHOIS the next day. Fantastic!

We order two more circuits. One from Time Warner, one from Verizon/MCI. Time Warner installs the circuit fairly easily. They give us a /30 and everything is good. Now we just need BGP.

So far I have filled out the Time Warner BGP request form three times. No response, not even an automated one. I have e-mailed tech support. No response, not even an automated one. I have called tech support. They say they can't do anything until I fill out the form. They say they have no requests from me. What gives?!?!? I'm giving up on them for now. At least until next week Monday. I don't give up easily.

Next we deal with Verizon. This has been interesting. Most people know of at least three Verizon-type companies:

- Verizon Wireless (cell phones)
- Verizon local (the ILEC)
- Verizon Business (used to be MCI, I guess)

So far, I have heard the following business names while trying to order/turn up this circuit:

- Verizon
- Verizon Legacy
- Verizon Core
- Verizon Business
- MCI
- UUNET

That's right: UUNET. Are you kidding me? Having six different names for your company is confusing enough. Using UUNET certainly doesn't help. Last time I heard UUNET it was the nineties and I was in middle school. I had to look it up on Wikipedia just to make sure she wasn't totally confused. Turns out it goes something like this:

UUNET -> MCI -> Verizon Business

So far their name hasn't been the only thing they are confused about. I don't even want to get into it right now. I'll make sure to update everyone as the Verizon/MCI/UUNET saga continues.

What is most surprising about all of this? Cogent. With it's horrible reputation and low cost half the people I talk to still cringe at the mention of the "C word". I can tell you this: they have been (by far) the easiest to deal with. Amazing. We'll see how the service is but as of now I am a happy Cogent customer. Anyone that would like to argue about them can try to deal with some of these other characters. Let me know how it goes!