Getting Multihomed - Part 3/3

Following up to one of my first posts. We FINALLY brought up BGP with all of our providers. A call from our CEO to some people at Verizon got some things moving again. I had the circuit up with BGP the same day. Pretty amazing, huh?

Anyways, now my problem was dealing with the limited memory and tcam allocation for unicast routes. If you recall, I ordered three full BGP feeds from three different providers. With the internet pushing 226,000 routes my 3750G wasn't going to cut it:

sh platform tcam utilization

CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values

Unicast mac addresses: 400/3200 13/44
IPv4 IGMP groups + multicast routes: 144/1152 6/26
IPv4 unicast directly-connected routes: 400/3200 13/44
IPv4 unicast indirectly-connected routes: 1040/8320 1023/8134
IPv4 policy based routing aces: 512/512 2/2
IPv4 qos aces: 512/512 8/8
IPv4 security aces: 1024/1024 23/23

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

So now I've got full feeds from three providers coming in. Luckily I read up on IOS route-map statements before I brought these BGP sessions up. Otherwise things could've gotten ugly. Here's what I started with:

ip as-path access-list 50 permit ^174$
ip as-path access-list 50 permit ^4323$
ip as-path access-list 50 permit ^701$

I started with just getting the ASNs we were directly connected to. And my tcam started to fill, but it wasn't close. I thought, hey, why not get some more routes while I can? I started to read up a bit more on route-maps and I figured out how to get other ASNs into my route table. I only want the networks of providers connected to my providers. Does that make sense?

Without being able to see the full table I would have no idea of what I was doing. What if I wanted Level(3)'s routes, for instance? I needed to see what was going on. Luckily an old client of mine runs FixedOrbit - the coolest site to look at BPG information. All I had to do was query my directly connected ASNs and start picking other routes I wanted. BGP would take care of the rest.

Here is a shortened version of what I ended up with:

ip as-path access-list 50 permit ^174$
ip as-path access-list 50 permit ^174_3356$
ip as-path access-list 50 permit ^174_33363$
ip as-path access-list 50 permit ^4323$
ip as-path access-list 50 permit ^4323_1668$
ip as-path access-list 50 permit ^4323_6983$
ip as-path access-list 50 permit ^4323_11456$
ip as-path access-list 50 permit ^701$
ip as-path access-list 50 permit ^701_19262$
ip as-path access-list 50 permit ^701_3356$

Now I have entries in my route table for my directly connected ASNs (174, 701, 4323) and some ASNs they are peered with - 3356, 33363, 1668, 6983, 11456, 19262. I don't have much room in my tcam but hey, that's what VXRs are for! Wow, I really want one of those (with an NPE-G2, of course) ;).

The Ultimate Geek Watch

I used to think watches were completely unnecessary and stupid. After all, my cellphone is always with me and is always synced to the right time. Why do I need a watch?

Then I found a white G-Shock. This watch has it all:

• It's white
• Atomic synced
• Shock resistant (tough)
• Vibration alarm
  
• Thermometer
• Solar Power
• World Time
• Movement sensor

I bought it a few months ago from amazon.com. Evidently it's imported by Mister Watch from Japan. Sure enough it came in the mail, complete with price tag (in yen) and a Japanese-only user's manual (G-Shock is made by Casio, a Japanese company).

Because I don't read Japanese it has been difficult to discover all of the features of this watch. The "movement sensor" that I described above is a good example. I was out with my friends one night in a dark bar and I lifted my wrist to look at the time. The back light automatically came on once I twisted my wrist a certain way. At first I thought it was a fluke. My friends were convinced I was somehow controlling it with my wrist, mind, etc. Oh no, it was the movement sensor.

I think it works in combination with the solar panel because it only activates when it is dark (beyond a certain point). In a dark enough room, with just the right wrist snap, I never have to manually push the light button to see what time (or temperature) it is. That's a good thing too, because like any good American, I don't want to have to do ANYTHING that I shouldn't have to do and pushing watch buttons is no exception.

It's too bad that you can't get them anymore because many, many people have asked about the white G-Shock. If anyone knows where you can get them in the US please let me know!

UPDATE: Shinya Amano has translated AstLinux documentation for voip-info.jp and he has done some research on this watch for me. He found the English manual. Thanks Amano!

GoDaddy Sucks

As I write this GoDaddy has managed to completely screw up authoritative DNS for krisk.org. They have been my registrar for quite some time. Yesterday I decided to switch my authoritative dns over to them from DynDns. Why would I do this?

I needed e-mail forwarding. KrisK.org used to be hosted on a FreeBSD server that I ran. I started to get more and more busy with AstLinux and other misc. stuff so I moved as many of my services to free or managed solutions.

I couldn't do e-mail forwarding because GoDaddy's recommended MX records were CNAMEs. DynDns does not approve of this (every MX should be an A record). So I needed to move everything to GoDaddy just so I could use their stupid free e-mail forwarding (krisk.org -> gmail).

I tried to get to this blog today and blog.krisk.org wasn't resolving. What gives? I tried digging a few DNS servers that I knew of. All of them returned NXDOMAIN. That's not good. I ran whois and krisk.org to find the authoritative name servers. I was (and still am) on ns5.secureserver.net and ns6.secureserver.net. I tried to do directly against them:

kris@krislap:~$ dig @ns5.secureserver.net

; <<>> DiG 9.3.4 <<>> @ns5.secureserver.net
; (1 server found)
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 11126
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;. IN NS

;; Query time: 70 msec
;; SERVER: 208.109.78.180#53(208.109.78.180)
;; WHEN: Wed Jul 25 15:22:43 2007
;; MSG SIZE rcvd: 17

Same deal with ns6. That's not good. Some of my other domains on GoDaddy have ns1 and ns2. I tried to dig against those and they worked. The funny thing was ns5 and ns6 were not found as NS records.

I shuddered at the thought of calling GoDaddy support. I certainly don't want to talk to any of "those people". By "those people" I mean script reading drones that would ask me which version of Internet Explorer I was using...

I logged into the extremely horrible GoDaddy portal and clicked "Use default hosting name servers". They were listed as ns5 and ns6. I wasn't hopeful. A few minutes later it appears to be working again:

kris@krislap:~$ dig @ns5.secureserver.net www.krisk.org

; <<>> DiG 9.3.4 <<>> @ns5.secureserver.net www.krisk.org
; (1 server found)
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11574
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;www.krisk.org. IN A

;; ANSWER SECTION:
www.krisk.org. 3600 IN CNAME godaddy.krisk.org.
godaddy.krisk.org. 3600 IN A 68.178.211.88

;; AUTHORITY SECTION:
krisk.org. 3600 IN NS NS1.SECURESERVER.NET.
krisk.org. 3600 IN NS NS2.SECURESERVER.NET.
krisk.org. 3600 IN NS ns5.SECURESERVER.NET.
krisk.org. 3600 IN NS ns6.SECURESERVER.NET.

;; Query time: 76 msec
;; SERVER: 208.109.78.180#53(208.109.78.180)
;; WHEN: Wed Jul 25 15:32:13 2007
;; MSG SIZE rcvd: 157


Woah! Look at that! ns1, ns2, ns5, and ns6 are listed as authority. Hmmm...

P.S. - If you are still reading this I need to tell you that DynDns is AWESOME!

AstLinux 0.4.7

AstLinux 0.4.7, with Asterisk 1.2.22 and Zaptel 1.2.19 has been released.

Finding Asterisk

I've run my own Asterisk system for quite some time. My mom lives in Minneapolis, my dad lives in Chicago and my sister lives in Montreal (stinkin' French Canadians - the worst kind). Just kidding, I love my sister and all of my Quebec friends! At this point in time I lived in Wisconsin. Cell phone plans weren't that cheap yet (especially in Canada) and my family was spread out far enough for every call to be long distance.

I started playing with Asterisk in July of 2004. It turns out that I was flying to Montreal to visit my sister during the US Independence Day Holiday (yeah, that makes sense - go to Canada for the 4th). I bought a cheap X100P clone, installed it in a Linux machine, and connected it to the analog line (SBC) at my house in Wisconsin.

I configured a basic (dangerous) Asterisk install over the course of a few days and packed my things for Montreal. Of course I also brought my laptop. I had also managed to install a softphone on it before I left. I got to my sister's apartment, connected my laptop to her VideoTron cable modem and registered with my softphone. Would it actually work? Could I make a call from this softphone to that Linux server (over the internet) and out the PSTN in Wisconsin to one of my friends - for free?

If you are reading this blog you probably know what happened next. It worked perfectly. I was amazed and my friends were impressed. This was awesome.

Over the next couple of weeks I added what I needed to my (lame) Asterisk configuration. Then disaster struck... My air conditioning broke and with it went the hard drive of the Asterisk play server. It was a play server - no backups.

I hate doing things twice. That's exactly how I felt about this. I wasn't going to work through all of that again - compiling, configuration file mess again. I swore I wasn't going to resurrect Asterisk or that machine again.

Then I heard it. Asterisk was calling. I couldn't get the idea of Asterisk out of my mind. I hit the books (voip-info.org, internet mostly) and I learned more about Asterisk - more than enough to simply rebuild my play server. A couple of days later I had it back up and better than before. In the process I also knew what I was going to do with it. I could ship an ATA to each member of my family and we could all call each other, for free. I could also get an account with an ITSP and provide cheap calling to my family members.

Today some instance of that server is still running and my family members can still call each other with a four digit dial, for free.

What's my name?

This is going to be a different kind of post. This post might actually be useful for people trying to solve this problem. Just the facts, ma'am.

One of the things that has repeatedly come up in my line of work is CallerID name delivery in PRI (Primary Rate Interface) ISDN (Integrated Services Digital Network) configurations. I learned more about CallerID name today than I ever wanted to know. Just kidding - I love getting into stuff like this!

PRI is great because call setup is fast and CallerID information is available instantly. Or is it? I always knew that Caller ID name is not carried over the PSTN (usually - in some countries it is). The number does (obviously), but the name is usually looked up in CNAM by the terminating switch, not the originating switch. What I didn't know is that sometimes this isn't done when the initial Q.931 Setup message comes down the PRI to signal a new call.

Sometimes this CNAM lookup takes a little while (fractions of a second) and the name is sent later in a separate Q.931 Facility message. This is true. Cisco says so (PDF). A Cisco ISDN-SIP gateway can be configured to do this one of two ways:

1) Wait until you receive the Q.931 Facility message with name and shove it into the SIP INVITE using either PAI (P-Asserted-Identity) or RPID (Remote-Party-ID). Send the INVITE to the SIP proxy (or wherever).

2) Send the INVITE ASAP, and then send a SIP INFO packet when the name shows up in the Q.931 Facility message.

The default is #2, which is screwy. Very cool, but still screwy. It is much harder to design a SIP platform that can accept the initial INVITE, begin to process the call, and then append the PAI or RPID information received in the later INFO.

Thanks to Cisco I now understand more about Q.931 and ISDN. Now I need to get this "thing" to work.

My test setup:

PRI -> Asterisk -> PRI -> AS5350XM -> SIP -> OpenSER -> SIP -> Device

I need to get Caller ID with name delivery through this whole mess, from the first PRI to the last SIP device.

The LEC provided the PRI coming into the Asterisk machine. I provided everything else. I saw several roadblocks:

1) Get the CID Name from the LEC (via PRI)
2) Pass it through Asterisk
3) Get it to the 5350 (via PRI)
4) Get it to OpenSER (via SIP)

Knowing what I now know about Caller ID with name in ISDN I knew just what to do for Asterisk. In zapata.conf, my incoming context is lec-in. Here it is (from extensions.conf):

[lec-in]

exten => NXXNXXXXXX,1,Wait(1)
exten => NXXNXXXXXX,n,DoSomethingElse

Yep, that's right. All you need is to Wait a little to get that second Facility IE. Asterisk doesn't support getting the Facility IE later and it certainly doesn't support sending a subsequent SIP INFO. That's a good thing because as I said the "other" way (SIP INFO) just seems goofy to me.

Now I needed to get the CallerID name to the 5350. It didn't seem to work. I start looking at "pri debug span 3" output to see the Q.931 goodness coming from Asterisk. I fired up "debug isdn q931" on the 5350. No dice. It looked like this bug in libpri was killing me:

http://bugs.digium.com/view.php?id=9651

This was committed to libpri SVN about a month ago. I update libpri from SVN, recompile Asterisk, and install the new chan_zap.so. I give it another shot. It looks like the 5350 is now getting the name over Q. 931. Using ngrep I look at the SIP INVITE coming into OpenSER from the 5350. I have an RPID header, but it looks strange. The name field in the Remote-Party-ID header is "pending". What the heck is that about? "pending" was not what I was seeing in Asterisk!

I opened up ngrep a bit to let my see any SIP INFO messages that might be coming later. Sure enough shortly after the SIP INVITE comes a SIP INFO message with my Caller ID name. Going back to my two configuration choices on the 5350 I knew I preferred option #1 (send everything in one SIP INVITE), even if it meant there was a little delay before the caller got audio. How could I configure the 5350 to wait a little and put it all in one SIP INVITE before the Cisco fired it off to OpenSER?

I dug around on cisco.com for a bit. Nothing - at least nothing obvious. You have to love Cisco configuration and Cisco docs. I decided to look around the internet and see if anyone else had this problem.

I looked on Google and found this:

http://puck.nether.net/pipermail/cisco-voip/2005-June/005485.html

I wondered if Mr. Adam Rothschild ever found the solution to his (my) problem. I open up another tab and write him an e-mail. Three minutes later (literally) he sends me this configuration snippet:

---Begin IOS Configuration---
interface Serial3/0:23
no ip address
load-interval 30
isdn switch-type primary-ni
isdn incoming-voice modem
isdn supp-service name calling
isdn negotiate-bchan
no isdn outgoing display-ie
no cdp enable
exit
gateway
timer receive-rtp 1200
sip-ua
disable-early-media 180
retry invite 3
retry response 3
retry bye 3
retry cancel 3
timers buffer-invite 500

---End IOS Configuration---

Let's get away from this technical mumbo-jumbo and talk about people for a minute...

Mr. Adam Rothschild got an e-mail from a random stranger across the internet referencing an obscure technical problem that he had over two years ago. In less than three minutes he dug up the solution and wrote me back. I have a SmartNet support contract on this 5350 but I doubt the techs at Cisco could have helped me any better or faster than a nice guy (Adam) helping a stranger (me).

Wipe away your tears, you sentimental fool. We're getting back to configuration. This blog is hardcore. Couldn't you tell?

I applied Adam's config to my AS5350XM running IOS 12.4(15)T. Here is the SIP INVITE from the 5350 to OpenSER:

U 192.168.0.1:61306 -> 192.168.0.10:5060
INVITE sip:9418675309@192.168.0.10:5060 SIP/2.0.
Via: SIP/2.0/UDP 192.168.0.1:5060;x-ds0num="ISDN 3/1:D 3/1:DS1
1:DS0";branch=z9hG4bK901AB9.
Remote-Party-ID: "STAR2STAR COMM"
;party=calling;screen=no;privacy=off.
From: "STAR2STAR COMM" ;tag=971D8C-1203.
To: .
Date: Mon, 16 Jul 2007 21:57:24 GMT.
Call-ID: 5E99C7DC-331E11DC-8126E6C7-399CBB13@192.168.0.1.
Supported: 100rel,timer,resource-priority,replaces.
Min-SE: 1800.
Cisco-Guid: 1586976572-857608668-2150694933-1673067056.
User-Agent: Cisco-SIPGateway/IOS-12.x.
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER,
SUBSCRIBE, NOTIFY, INFO, REGISTER.
CSeq: 101 INVITE.
Max-Forwards: 70.
Timestamp: 1184623044.
Contact: .
Expires: 300.
Allow-Events: telephone-event.
Content-Type: application/sdp.
Content-Disposition: session;handling=required.
Content-Length: 288.
.
v=0.
o=CiscoSystemsSIP-GW-UserAgent 7275 8957 IN IP4 192.168.0.1.
s=SIP Call.
c=IN IP4 192.168.0.1.
t=0 0.
m=audio 20746 RTP/AVP 18 0 101.
c=IN IP4 192.168.0.1.
a=rtpmap:18 G729/8000.
a=fmtp:18 annexb=no.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16

Yeah yeah! Look at that Caller ID name in that Remote-Party-ID header! I feel like that's the best looking SIP INVITE I have ever seen. How does one SIP INVITE look better than any other? If you don't know the answer to that question, you haven't been following along.

I wrote Adam back to let him know how it turned out. He wrote me back again, happy to hear that it worked for me. Wow, just wow.

So many things shine through in this post. In one evening I found (and patched) a bug in libpri. I learned more about Q.931 and Caller ID. I found a guy to help me put it all together. The open source development model worked. The promise of easy access to information via the internet skooled me in ISDN. Social networking proved to be very effective, even while using pre-web 2.0 technology (e-mail). Google worked (a lot).

Now I get to put it all together in this blog post to give back a little. Hopefully the next guy (or girl) trying to get some mixed up mess of SIP and ISDN devices to work together with Caller ID Name delivery will get out of the office just a little bit earlier.

AdSense

It has been three days since my first post. I feel like I have done a pretty good job letting Google/AdSense know what this blog is going to be about. I've mentioned Cisco, Asterisk, AstLinux, trixbox, BGP, etc. What gives AdSense? I'd love to specifically mention the ridiculous things you are trying to sell my readers - but I won't. I won't out of fear that you will pick up on this post and not the others, sinking me even deeper into non-relevant AdSense hell. The ads on AstLinux.org are perfect. Give me some of that!