What a difference a day makes!

DISCLAIMER - I've been sick for a couple of weeks and after reading my last post, I feel I need to warn you that my writing quality will be sub par until I get over this.... Clearly I'm not running at %100!

Yesterday I was ranting about my frustrations with modern technology. Today I'll be making some apologies.

I wrote my post yesterday after struggling with a Snom 360 for a couple of hours. I came at it today, fresh from my five hours of sleep last night and got it working in 10 minutes. A long time ago while working my first real job at Wisconsin Vision Associates, my co-worker taught me the benefits of a "fresh pair of eyes". I had a tendency to obsessively attack a problem until I solved it. This had been an issue of mine for quite some time. I rarely made it to my first class during high school because I would often stay up until 6am working on some computer problem I was facing. Dave used to remind me to stop and try again later with a fresh pair of eyes.

Looks like that's all I needed. I came in today, tried upgrading the firmware (using the bootloader interrupt method) and the Snom 360 came to life. I had a couple of reasons for testing the Snom 360. First, I need to configure them for Star2Star. Secondly, I wanted to test the Asterisk sip-tlstcp branch. Why TCP for SIP? Why TLS? Why bother?

First of all, SIP is pretty insecure. It's 2007. Everything, everywhere should be encrypted. There's just no excuse for it anymore. People transport some pretty valuable data over the telephone. Their credit card numbers, their banking/identity/medical details, their deepest, most private thoughts. Not the kind of stuff you want just anyone to get their hands on.

If you are using standard VoIP, it's probably SIP. SIP is a signalling protocol, it handles session initiation (hey - Session Initiation Protocol) for media sessions that (typically, but don't have to) use RTP (Realtime Transport Protocol). In a typical call scenario, the source and destination phone numbers, caller name, etc would be transported by SIP. While this data isn't terribly important, it could be valuable to an attacker. Most of the good stuff, however, is transported using RTP. This includes DTMF digits (if using inband or RFC2833 signaling) and audio (your voice, music on hold, heavy breathing, whatever). It's more important to secure RTP. The best standard for this is currently SRTP (Secure RTP). The problem is, like all crypto, there has to be an exchange of keys. How do we do this? Easy - SIP. But what if the SIP channel isn't secure? Ah ha. It needs to be. That's where SIP TLS comes in.

Fine, I'll implement SIP TLS then. Well, in Asterisk you have another problem. TLS (usually) requires TCP. The SIP channel driver in Asterisk doesn't support TCP. Why use TCP for SIP? Isn't that bad, doesn't that mean my voice packets will be delayed/blow up/melt my router? Shame on you! You haven't been paying attention. Your voice uses RTP and even with SIP TCP/TLS that's still UDP (even with SRTP). Don't worry about that. Only the session-type stuff (SIP) will use TCP. Why bother with TCP? Well, first of all, it's cool. Secondly, we get to use TLS. Third, TCP allows for packet fragmentation and (ultimately) will allow for more cool stuff to happen. For instance, if you've got some crazy videophone that supports a million types of sessions, codecs, etc your SIP+SDP infoz could possibly exceed the MTU size of your connection. Bad things will happen.

With this branch of Asterisk, we get all of that cool stuff above with the exception of the actual SRTP implementation. Don't get so greedy! I'm sure it's coming. This is open source, after all. If you don't like it you can ask for your money back and deal with some other vendors.

Anyways, back to the original point. I wanted to test this branch and it had already been tested with all of the other equipment Star2Star uses - Cisco gateways, Polycom phones, other Asterisk systems, etc. I wondered if I could get these Snom phones to work with it. But first I had to get them working with Asterisk and standard SIP/UDP. I did, and a little while later, I got it the Snom to use TCP/TLS. From a message I posted to Asterisk-dev:

*CLI> core show version
Asterisk SVN-group-sip-tcptls-r92242-

/trunk built by kris @ krislap on
a i686 running Linux on 2007-12-10 19:29:26 UTC
*CLI> sip show peer snom


* Name : snom
Secret :
MD5Secret :
Context : default
Subscr.Cont. :
Language :
AMA flags : Unknown
Transfer mode: open
CallingPres : Presentation Allowed, Not Screened
Callgroup :
Pickupgroup :
Mailbox :
VM Extension : asterisk
LastMsgsSent : 32767/65535
Call limit : 0
Dynamic : Yes
Callerid : "" <>
MaxCallBR : 384 kbps
Expire : 3028
Insecure : no
Nat : RFC3581
ACL : No
T38 pt UDPTL : No
CanReinvite : Yes
PromiscRedir : No
User=Phone : No
Video Support: Yes
Text Support : No
Trust RPID : Yes
Send RPID : Yes
Subscriptions: Yes
Overlap dial : No
DTMFmode : rfc2833
ToHost :
Addr->IP : 10.16.5.237 Port 2060
Defaddr->IP : 0.0.0.0 Port 5060
Transport : TLS
Def. Username: snom
SIP Options : (none)
Codecs : 0x4 (ulaw)
Codec Order : (ulaw:20)
Auto-Framing: No
100 on REG : No
Status : OK (25 ms)
Useragent : snom360/7.1.30
Reg. Contact : sip:snom@10.16.5.237:2060;transport=tls;line=1sz3a8qe

*CLI> sip show tcp
Host Port Transport Type
10.16.5.237 2077 TLS Server

Not only did I get this phone to work, I got it to work with some super-cool bleeding version of Asterisk. Heck yeah. Plus, it turns out I really like Snom phones. Maybe even more than Polycom!

I'd like to clear the air about something else too... I was really angry at BMW yesterday. So angry that I made an appointment to test drive a potential new car - a Maserati Quattroporte. It's an Italian thoroughbred. With a Ferrari engine, Pininfarina styling, and a legendary name to boot. A few miles down the road, I realized BMW was the car for me. Don't get me wrong, the Maserati is a really nice car but it has some serious shortcomings. The GPS system looks like 8-bit Nintendo. The DuoSelect transmission revs WAY to high and is too clunky. Two serious problems.

In summary, I have some apologies to make - to Snom, to BMW, and to an entire country - Germany. Over-engineered or not, you're still the best at it.

Modern Technology

VoIP really bothers me sometimes... Why is it that even after YEARS of dealing with this stuff do some things just seem to be ridiculously complicated? For instance - today a Snom 360 arrived. My goal is to get this thing ready to integrate with Star2Star. That means:

• Remote firmware upgrades
• No (little) touch provisioning
• Speed dials, monitoring, etc

I've had this thing on my desk for a little over an hour and the first requirement (firmware upgrade) cannot be met because the damn things HTTP/HTTPS server disappears a few seconds after the phone boots up. WTF? Yes, I am working on this now, I am writing this now, and I am angry now. I've been working with this stuff for years and I am AMAZED it still takes this long to get a phone working. Call this progress (no pun intended)? I don't think so. Fifty years ago (if I were alive, I suppose) I could go buy any analog phone, plug it in, and carry on with my life. Instead I'm wasting it away with this phone/computer Frankenstein sitting on my desk.

Reminds me of my car (also German - BMW). About a month ago the remotes just stopped working. After taking it in a few times over the course of a couple of weeks, they FINALLY figured out what was wrong. They replaced almost $1500 worth of parts (still under warranty, thank God) and spent days (literally) "upgrading and rebooting" various computer and software components to ensure compatibility with the new hardware. I get the car back and the computer had been completely re-initialized. Everything needs to be replaced and reprogrammed. Even after setting it up again, my Nokia E-70's bluetooth didn't work with the car. It is paired and recognized but any call results in no audio - makes it kind of tough to talk "hands free". Of course it worked quite well before the software upgrade...

Now I'm trying to figure out why the web server on the Snom keeps disappearing. Is it a bug (running firmware 7.1.8)? Some kind of "feature" (another example of German over-engineering, perhaps)? At the moment I'm leaning towards bug because this thing has got some other really interesting quirks... I changed the VLAN setting, rebooted, and still had the DHCP address from the original VLAN but it wasn't reachable. The phone had joined the new VLAN but did not obtain a new DHCP address. If this were in the field, this phone would be bricked (from a network perspective). If this were Grandstream I would understand (expect) this. From someone with a good reputation like Snom it is very disappointing!

VoIP, Bluetooth, Snom, BMW, Nokia. Are our lives REALLY any better?

Monitoring BGP Feeds

Two weeks! I can't believe it has been that long since my last post. That's just crazy!

I know that I can't make it up to you but I will share something I just finished up.

BGP is cool. It is so cool to turn it up on a new router and see the entire internet's routing table with a simple command "sh ip bgp". What's cooler than that?

Feeding that table into a database, that's what!

Putting your BGP feed into a database enables all sorts of cool things. As a matter of fact, some of these are so cool I haven't even thought of them yet! For the last hour or so I've been busy working on getting this going. Here's what you will need:

- BGP feed from an upstream provider, connected to a router
- Linux machine running Quagga
- Linux machine with Perl installed (can be same machine as Quagga, mine is)

First you will need to configure your router with BGP enabled:

en

conf t

ip as-path access-list 1 deny .*

router bgp [your ASN]
neighbor [Quagga IP] remote-as 64512
neighbor [Quagga IP] transport connection-mode passive
neighbor [Quagga IP] description Quagga peer
neighbor [Quagga IP] filter-list 1 in


You will want to make sure that this machine is directly connected. If it isn't you need multihop BGP (which I won't cover right now). Here's what we're doing:

- The as-path with deny updates from your Quagga machine. We don't want some misconfiguration to actually affect your network. We just want some routes from the Cisco!

- Create the neighbor with remote-as 64512 (private ASN)

- Don't initiate a connection to this peer, let them connect to us (passive)

- Apply the filter-list to inbound traffic for this neighbor (don't allow updates from Quagga)

Now we need to configure Quagga. First, you will need to install it. Your distribution should have some packages for you. Use yum, apt, etc to grab it.

It will probably install some config files in /etc/quagga. We only want to setup bgp. This should be a good sample bgpd.conf to get you started:

hostname [your hostname]
password changeme
enable password changeme
log stdout
log syslog
service advanced-vty
!
router bgp 64512
bgp router-id [Quagga IP]
neighbor [Cisco IP] remote-as [your real ASN]
neighbor [Cisco IP] description Internet BGP Feed
neighbor 127.0.0.1 remote-as 64512
neighbor 127.0.0.1 description local db hookup
neighbor 127.0.0.1 port 9179
neighbor 127.0.0.1 passive
neighbor 127.0.0.1 filter-list 1 in
neighbor 127.0.0.1 next-hop-self
neighbor [some public ip] remote-as 64513
neighbor [some public ip] description Remote devel
neighbor [some public ip] passive
neighbor [some public ip] ebgp-multihop 255
neighbor [some public ip] filter-list 1 in
neighbor [some public ip] next-hop-self
!
access-list 1 permit [local class C network] 0.0.0.255
access-list 1 permit 127.0.0.1
access-list 1 deny any
access-list 10 permit [local class C network] 0.0.0.255
access-list 10 permit [remote ip]
access-list 10 deny any
!
ip as-path access-list 1 deny .*
!
line vty
access-class 1


After you apply this, you will want to start bgpd: "bgpd -n". It will tell you which vty you can connect to with telnet:

telnet localhost 2605

That should work. At this point, you should have a connection up to your main router:

sh ip bgp sum
BGP router identifier (deleted), local AS number 64512
29 BGP AS-PATH entries
0 BGP community entries

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
(deleted) 4 15092 580 129 0 0 0 02:06:29 9845
127.0.0.1 4 64512 167 944 0 0 0 00:58:59 0
(deleted) 4 64513 6 616 0 0 0 00:02:56 0

Total number of neighbors 3

Sweet! Hey, where did that connection from 127.0.0.1 come from? That looks strange...

That connection is the whole point of this exercise. Star2Star is interested in quality, internet routing and the relationship between the two (at least I am). Long ago we realized that having a full BGP feed and being able to analyze updates in real time could be a huge asset to our management and monitoring portfolio.

How could we do this? Within the last week it came up on NANOG. A guy named Bill Nash wrote a Perl script a while back using Net::BGP to connect to a BGP peer and dump it's routes into a MySQL database. This made much more sense than trying to pull them via SNMP or some other hackish mechanism.

Bill wrote this for another employer and no longer had access to it. Evidently enough people asked him about it off list for him to consider re-writing it. I wanted him to do more than consider... I wanted that script!

I contacted Bill and offered to support his work any way possible. This included setting up a read-only BGP feed for him. I didn't want to give him (no offense Bill) a direct connection to our Cisco router (crashing that would be BAD) so I came up with the setup above. The Quagga is read only (by configuration and with -n on the command line not even the local kernel can be updated). And the Cisco is read-only to the Quagga peer. Seems safe enough.

The Quagaa instance is merely a distributor for our BGP feed. That way I can mess with it all I want without any fear (or very little fear) of causing any problems for our main router. I can hammer it all I want with some alpha-quality perl scripts. Worst case (hopefully) I'll just hose Quagga if something goes wrong...

While waiting for Bill to get his Perl script going, I Googled BGP Perl to see if there was anything else out there. Sure enough, there is:

http://briangannon.com/2007/04/23/bgp-perl-route-analyzer/

This is a crude version of what I am looking for. I made a few minor changes because I needed it to run on the same machine as Quagga (already using TCP port 179). I also didn't want to have to run the perl script as root. Here is a mini-diff:

line 20:
-my $bgp = new Net::BGP::Process();
+my $bgp = new Net::BGP::Process( Port => 9179 );

Now when you follow the directions on Brian's blog to INSERT the BGP peer into the SQL table, make sure to just use localhost. Then the perl script will use port 9179 for itself. After all, if nothing needs to connect to it, who cares what the local port is (as long as the peer has been configured properly)? Quagga knows that peer 127.0.0.1 is on port 9179, and it works. Check this out:

6147167 | 4 | 216.134.176.0/22 | 2007-08-28 19:39:51 | 2 | Next Hop Changed,Metric Changed |
| 6147168 | 4 | 216.134.180.0/22 | 2007-08-28 19:39:51 | 2 | Next Hop Changed,Metric Changed |
| 6147169 | 4 | 216.85.83.0/24 | 2007-08-28 19:39:51 | 0 | Removal of network |
| 6147170 | 4 | 216.85.83.0/24 | 2007-08-28 19:39:57 | 1 | Added 216.85.83.0/24 |
| 6147171 | 4 | 216.85.83.0/24 | 2007-08-28 19:40:35 | 0 | Removal of network |
| 6147172 | 4 | 216.85.83.0/24 | 2007-08-28 19:41:22 | 1 | Added 216.85.83.0/24 |
| 6147173 | 4 | 207.250.244.0/23 | 2007-08-28 19:48:32 | 1 | Added 207.250.244.0/23

or the routes:

mysql> select * from route limit 9842,100;
+---------+-----------+------------------+-----------+--------+------------+---------------+
| id | router_id | prefix | next_hop | metric | local_pref | as_path |
+---------+-----------+------------------+-----------+--------+------------+---------------+
| 3045536 | 4 | 216.85.190.0/24 | 127.0.0.1 | | 100 | 15092 4323 |
| 3045638 | 4 | 198.102.2.0/24 | 127.0.0.1 | | 100 | 15092 4323 |
| 3045655 | 4 | 195.85.117.0/24 | 127.0.0.1 | | 100 | 15092 174 209 |
| 3045665 | 4 | 216.85.83.0/24 | 127.0.0.1 | 99999 | 100 | 15092 4323 |
| 3045666 | 4 | 207.250.244.0/23 | 127.0.0.1 | 99999 | 100 | 15092 4323 |
+---------+-----------+------------------+-----------+--------+------------+---------------+


That's from my MySQL db. Pretty cool, huh?

I'll be working more on this in the upcoming weeks. Bill will also be working on a much improved version of the BGP Perl script that I am working with now. I'll make sure to let everyone know how it goes!

Social Networking

As of today I rounded out my social networking portfolio.

A year ago I didn't belong to any of these "web 2.0"/"social networking" sites. First it was Orkut. I was in Brazil and it kept coming up. Why not? So I joined.

A couple months ago my friends in Sarasota, FL kept started really bothering me about MySpace. Why not? So I joined.

A couple of days ago I got enough LinkedIn invitations for me to break down and create an account.

Today I signed up for Facebook, much to my chagrin. I was already on three other sites, so why not?

I'll tell you why not. Now I am going to have people complaining about my outdated profiles, lack of interest, etc. Why create an account if you can't keep it up to date?

How am I supposed to maintain accounts on my personal/professional life spread across FOUR different social networking sites?!? This is madness. I can't wait to see what everything looks like in a few months...

So anyways if you are on any of these sites you should try to track me down to see how it all unfolds. I am sure it will be interesting!

Update!

I am still alive - barely.

I haven't been able to post over the last couple of weeks because Star2Star was busy getting another release out. We put out another release every six months (depending on schedule and delays) and you guessed it - it's that time of the year again.

Our latest release is 2.1. It includes a lot of fixes and feature improvements to the overall system, everything from Polycom firmware to OpenSER enhancements (lots of them).

Speaking of OpenSER, it looks like I will be working with it quite a bit over the next few weeks and months for the 2.2 release. I'll also have some interesting Cisco experiences, I'm sure...

So between getting this release out, a car accident, and regular life I have not had much time for this blog. Things should be getting back to normal pretty soon. I like it that way.

Getting Multihomed - Part 3/3

Following up to one of my first posts. We FINALLY brought up BGP with all of our providers. A call from our CEO to some people at Verizon got some things moving again. I had the circuit up with BGP the same day. Pretty amazing, huh?

Anyways, now my problem was dealing with the limited memory and tcam allocation for unicast routes. If you recall, I ordered three full BGP feeds from three different providers. With the internet pushing 226,000 routes my 3750G wasn't going to cut it:

sh platform tcam utilization

CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values

Unicast mac addresses: 400/3200 13/44
IPv4 IGMP groups + multicast routes: 144/1152 6/26
IPv4 unicast directly-connected routes: 400/3200 13/44
IPv4 unicast indirectly-connected routes: 1040/8320 1023/8134
IPv4 policy based routing aces: 512/512 2/2
IPv4 qos aces: 512/512 8/8
IPv4 security aces: 1024/1024 23/23

Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization

So now I've got full feeds from three providers coming in. Luckily I read up on IOS route-map statements before I brought these BGP sessions up. Otherwise things could've gotten ugly. Here's what I started with:

ip as-path access-list 50 permit ^174$
ip as-path access-list 50 permit ^4323$
ip as-path access-list 50 permit ^701$

I started with just getting the ASNs we were directly connected to. And my tcam started to fill, but it wasn't close. I thought, hey, why not get some more routes while I can? I started to read up a bit more on route-maps and I figured out how to get other ASNs into my route table. I only want the networks of providers connected to my providers. Does that make sense?

Without being able to see the full table I would have no idea of what I was doing. What if I wanted Level(3)'s routes, for instance? I needed to see what was going on. Luckily an old client of mine runs FixedOrbit - the coolest site to look at BPG information. All I had to do was query my directly connected ASNs and start picking other routes I wanted. BGP would take care of the rest.

Here is a shortened version of what I ended up with:

ip as-path access-list 50 permit ^174$
ip as-path access-list 50 permit ^174_3356$
ip as-path access-list 50 permit ^174_33363$
ip as-path access-list 50 permit ^4323$
ip as-path access-list 50 permit ^4323_1668$
ip as-path access-list 50 permit ^4323_6983$
ip as-path access-list 50 permit ^4323_11456$
ip as-path access-list 50 permit ^701$
ip as-path access-list 50 permit ^701_19262$
ip as-path access-list 50 permit ^701_3356$

Now I have entries in my route table for my directly connected ASNs (174, 701, 4323) and some ASNs they are peered with - 3356, 33363, 1668, 6983, 11456, 19262. I don't have much room in my tcam but hey, that's what VXRs are for! Wow, I really want one of those (with an NPE-G2, of course) ;).

The Ultimate Geek Watch

I used to think watches were completely unnecessary and stupid. After all, my cellphone is always with me and is always synced to the right time. Why do I need a watch?

Then I found a white G-Shock. This watch has it all:

• It's white
• Atomic synced
• Shock resistant (tough)
• Vibration alarm
  
• Thermometer
• Solar Power
• World Time
• Movement sensor

I bought it a few months ago from amazon.com. Evidently it's imported by Mister Watch from Japan. Sure enough it came in the mail, complete with price tag (in yen) and a Japanese-only user's manual (G-Shock is made by Casio, a Japanese company).

Because I don't read Japanese it has been difficult to discover all of the features of this watch. The "movement sensor" that I described above is a good example. I was out with my friends one night in a dark bar and I lifted my wrist to look at the time. The back light automatically came on once I twisted my wrist a certain way. At first I thought it was a fluke. My friends were convinced I was somehow controlling it with my wrist, mind, etc. Oh no, it was the movement sensor.

I think it works in combination with the solar panel because it only activates when it is dark (beyond a certain point). In a dark enough room, with just the right wrist snap, I never have to manually push the light button to see what time (or temperature) it is. That's a good thing too, because like any good American, I don't want to have to do ANYTHING that I shouldn't have to do and pushing watch buttons is no exception.

It's too bad that you can't get them anymore because many, many people have asked about the white G-Shock. If anyone knows where you can get them in the US please let me know!

UPDATE: Shinya Amano has translated AstLinux documentation for voip-info.jp and he has done some research on this watch for me. He found the English manual. Thanks Amano!

GoDaddy Sucks

As I write this GoDaddy has managed to completely screw up authoritative DNS for krisk.org. They have been my registrar for quite some time. Yesterday I decided to switch my authoritative dns over to them from DynDns. Why would I do this?

I needed e-mail forwarding. KrisK.org used to be hosted on a FreeBSD server that I ran. I started to get more and more busy with AstLinux and other misc. stuff so I moved as many of my services to free or managed solutions.

I couldn't do e-mail forwarding because GoDaddy's recommended MX records were CNAMEs. DynDns does not approve of this (every MX should be an A record). So I needed to move everything to GoDaddy just so I could use their stupid free e-mail forwarding (krisk.org -> gmail).

I tried to get to this blog today and blog.krisk.org wasn't resolving. What gives? I tried digging a few DNS servers that I knew of. All of them returned NXDOMAIN. That's not good. I ran whois and krisk.org to find the authoritative name servers. I was (and still am) on ns5.secureserver.net and ns6.secureserver.net. I tried to do directly against them:

kris@krislap:~$ dig @ns5.secureserver.net

; <<>> DiG 9.3.4 <<>> @ns5.secureserver.net
; (1 server found)
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 11126
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;. IN NS

;; Query time: 70 msec
;; SERVER: 208.109.78.180#53(208.109.78.180)
;; WHEN: Wed Jul 25 15:22:43 2007
;; MSG SIZE rcvd: 17

Same deal with ns6. That's not good. Some of my other domains on GoDaddy have ns1 and ns2. I tried to dig against those and they worked. The funny thing was ns5 and ns6 were not found as NS records.

I shuddered at the thought of calling GoDaddy support. I certainly don't want to talk to any of "those people". By "those people" I mean script reading drones that would ask me which version of Internet Explorer I was using...

I logged into the extremely horrible GoDaddy portal and clicked "Use default hosting name servers". They were listed as ns5 and ns6. I wasn't hopeful. A few minutes later it appears to be working again:

kris@krislap:~$ dig @ns5.secureserver.net www.krisk.org

; <<>> DiG 9.3.4 <<>> @ns5.secureserver.net www.krisk.org
; (1 server found)
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11574
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;www.krisk.org. IN A

;; ANSWER SECTION:
www.krisk.org. 3600 IN CNAME godaddy.krisk.org.
godaddy.krisk.org. 3600 IN A 68.178.211.88

;; AUTHORITY SECTION:
krisk.org. 3600 IN NS NS1.SECURESERVER.NET.
krisk.org. 3600 IN NS NS2.SECURESERVER.NET.
krisk.org. 3600 IN NS ns5.SECURESERVER.NET.
krisk.org. 3600 IN NS ns6.SECURESERVER.NET.

;; Query time: 76 msec
;; SERVER: 208.109.78.180#53(208.109.78.180)
;; WHEN: Wed Jul 25 15:32:13 2007
;; MSG SIZE rcvd: 157


Woah! Look at that! ns1, ns2, ns5, and ns6 are listed as authority. Hmmm...

P.S. - If you are still reading this I need to tell you that DynDns is AWESOME!

AstLinux 0.4.7

AstLinux 0.4.7, with Asterisk 1.2.22 and Zaptel 1.2.19 has been released.

Finding Asterisk

I've run my own Asterisk system for quite some time. My mom lives in Minneapolis, my dad lives in Chicago and my sister lives in Montreal (stinkin' French Canadians - the worst kind). Just kidding, I love my sister and all of my Quebec friends! At this point in time I lived in Wisconsin. Cell phone plans weren't that cheap yet (especially in Canada) and my family was spread out far enough for every call to be long distance.

I started playing with Asterisk in July of 2004. It turns out that I was flying to Montreal to visit my sister during the US Independence Day Holiday (yeah, that makes sense - go to Canada for the 4th). I bought a cheap X100P clone, installed it in a Linux machine, and connected it to the analog line (SBC) at my house in Wisconsin.

I configured a basic (dangerous) Asterisk install over the course of a few days and packed my things for Montreal. Of course I also brought my laptop. I had also managed to install a softphone on it before I left. I got to my sister's apartment, connected my laptop to her VideoTron cable modem and registered with my softphone. Would it actually work? Could I make a call from this softphone to that Linux server (over the internet) and out the PSTN in Wisconsin to one of my friends - for free?

If you are reading this blog you probably know what happened next. It worked perfectly. I was amazed and my friends were impressed. This was awesome.

Over the next couple of weeks I added what I needed to my (lame) Asterisk configuration. Then disaster struck... My air conditioning broke and with it went the hard drive of the Asterisk play server. It was a play server - no backups.

I hate doing things twice. That's exactly how I felt about this. I wasn't going to work through all of that again - compiling, configuration file mess again. I swore I wasn't going to resurrect Asterisk or that machine again.

Then I heard it. Asterisk was calling. I couldn't get the idea of Asterisk out of my mind. I hit the books (voip-info.org, internet mostly) and I learned more about Asterisk - more than enough to simply rebuild my play server. A couple of days later I had it back up and better than before. In the process I also knew what I was going to do with it. I could ship an ATA to each member of my family and we could all call each other, for free. I could also get an account with an ITSP and provide cheap calling to my family members.

Today some instance of that server is still running and my family members can still call each other with a four digit dial, for free.

What's my name?

This is going to be a different kind of post. This post might actually be useful for people trying to solve this problem. Just the facts, ma'am.

One of the things that has repeatedly come up in my line of work is CallerID name delivery in PRI (Primary Rate Interface) ISDN (Integrated Services Digital Network) configurations. I learned more about CallerID name today than I ever wanted to know. Just kidding - I love getting into stuff like this!

PRI is great because call setup is fast and CallerID information is available instantly. Or is it? I always knew that Caller ID name is not carried over the PSTN (usually - in some countries it is). The number does (obviously), but the name is usually looked up in CNAM by the terminating switch, not the originating switch. What I didn't know is that sometimes this isn't done when the initial Q.931 Setup message comes down the PRI to signal a new call.

Sometimes this CNAM lookup takes a little while (fractions of a second) and the name is sent later in a separate Q.931 Facility message. This is true. Cisco says so (PDF). A Cisco ISDN-SIP gateway can be configured to do this one of two ways:

1) Wait until you receive the Q.931 Facility message with name and shove it into the SIP INVITE using either PAI (P-Asserted-Identity) or RPID (Remote-Party-ID). Send the INVITE to the SIP proxy (or wherever).

2) Send the INVITE ASAP, and then send a SIP INFO packet when the name shows up in the Q.931 Facility message.

The default is #2, which is screwy. Very cool, but still screwy. It is much harder to design a SIP platform that can accept the initial INVITE, begin to process the call, and then append the PAI or RPID information received in the later INFO.

Thanks to Cisco I now understand more about Q.931 and ISDN. Now I need to get this "thing" to work.

My test setup:

PRI -> Asterisk -> PRI -> AS5350XM -> SIP -> OpenSER -> SIP -> Device

I need to get Caller ID with name delivery through this whole mess, from the first PRI to the last SIP device.

The LEC provided the PRI coming into the Asterisk machine. I provided everything else. I saw several roadblocks:

1) Get the CID Name from the LEC (via PRI)
2) Pass it through Asterisk
3) Get it to the 5350 (via PRI)
4) Get it to OpenSER (via SIP)

Knowing what I now know about Caller ID with name in ISDN I knew just what to do for Asterisk. In zapata.conf, my incoming context is lec-in. Here it is (from extensions.conf):

[lec-in]

exten => NXXNXXXXXX,1,Wait(1)
exten => NXXNXXXXXX,n,DoSomethingElse

Yep, that's right. All you need is to Wait a little to get that second Facility IE. Asterisk doesn't support getting the Facility IE later and it certainly doesn't support sending a subsequent SIP INFO. That's a good thing because as I said the "other" way (SIP INFO) just seems goofy to me.

Now I needed to get the CallerID name to the 5350. It didn't seem to work. I start looking at "pri debug span 3" output to see the Q.931 goodness coming from Asterisk. I fired up "debug isdn q931" on the 5350. No dice. It looked like this bug in libpri was killing me:

http://bugs.digium.com/view.php?id=9651

This was committed to libpri SVN about a month ago. I update libpri from SVN, recompile Asterisk, and install the new chan_zap.so. I give it another shot. It looks like the 5350 is now getting the name over Q. 931. Using ngrep I look at the SIP INVITE coming into OpenSER from the 5350. I have an RPID header, but it looks strange. The name field in the Remote-Party-ID header is "pending". What the heck is that about? "pending" was not what I was seeing in Asterisk!

I opened up ngrep a bit to let my see any SIP INFO messages that might be coming later. Sure enough shortly after the SIP INVITE comes a SIP INFO message with my Caller ID name. Going back to my two configuration choices on the 5350 I knew I preferred option #1 (send everything in one SIP INVITE), even if it meant there was a little delay before the caller got audio. How could I configure the 5350 to wait a little and put it all in one SIP INVITE before the Cisco fired it off to OpenSER?

I dug around on cisco.com for a bit. Nothing - at least nothing obvious. You have to love Cisco configuration and Cisco docs. I decided to look around the internet and see if anyone else had this problem.

I looked on Google and found this:

http://puck.nether.net/pipermail/cisco-voip/2005-June/005485.html

I wondered if Mr. Adam Rothschild ever found the solution to his (my) problem. I open up another tab and write him an e-mail. Three minutes later (literally) he sends me this configuration snippet:

---Begin IOS Configuration---
interface Serial3/0:23
no ip address
load-interval 30
isdn switch-type primary-ni
isdn incoming-voice modem
isdn supp-service name calling
isdn negotiate-bchan
no isdn outgoing display-ie
no cdp enable
exit
gateway
timer receive-rtp 1200
sip-ua
disable-early-media 180
retry invite 3
retry response 3
retry bye 3
retry cancel 3
timers buffer-invite 500

---End IOS Configuration---

Let's get away from this technical mumbo-jumbo and talk about people for a minute...

Mr. Adam Rothschild got an e-mail from a random stranger across the internet referencing an obscure technical problem that he had over two years ago. In less than three minutes he dug up the solution and wrote me back. I have a SmartNet support contract on this 5350 but I doubt the techs at Cisco could have helped me any better or faster than a nice guy (Adam) helping a stranger (me).

Wipe away your tears, you sentimental fool. We're getting back to configuration. This blog is hardcore. Couldn't you tell?

I applied Adam's config to my AS5350XM running IOS 12.4(15)T. Here is the SIP INVITE from the 5350 to OpenSER:

U 192.168.0.1:61306 -> 192.168.0.10:5060
INVITE sip:9418675309@192.168.0.10:5060 SIP/2.0.
Via: SIP/2.0/UDP 192.168.0.1:5060;x-ds0num="ISDN 3/1:D 3/1:DS1
1:DS0";branch=z9hG4bK901AB9.
Remote-Party-ID: "STAR2STAR COMM"
;party=calling;screen=no;privacy=off.
From: "STAR2STAR COMM" ;tag=971D8C-1203.
To: .
Date: Mon, 16 Jul 2007 21:57:24 GMT.
Call-ID: 5E99C7DC-331E11DC-8126E6C7-399CBB13@192.168.0.1.
Supported: 100rel,timer,resource-priority,replaces.
Min-SE: 1800.
Cisco-Guid: 1586976572-857608668-2150694933-1673067056.
User-Agent: Cisco-SIPGateway/IOS-12.x.
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER,
SUBSCRIBE, NOTIFY, INFO, REGISTER.
CSeq: 101 INVITE.
Max-Forwards: 70.
Timestamp: 1184623044.
Contact: .
Expires: 300.
Allow-Events: telephone-event.
Content-Type: application/sdp.
Content-Disposition: session;handling=required.
Content-Length: 288.
.
v=0.
o=CiscoSystemsSIP-GW-UserAgent 7275 8957 IN IP4 192.168.0.1.
s=SIP Call.
c=IN IP4 192.168.0.1.
t=0 0.
m=audio 20746 RTP/AVP 18 0 101.
c=IN IP4 192.168.0.1.
a=rtpmap:18 G729/8000.
a=fmtp:18 annexb=no.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16

Yeah yeah! Look at that Caller ID name in that Remote-Party-ID header! I feel like that's the best looking SIP INVITE I have ever seen. How does one SIP INVITE look better than any other? If you don't know the answer to that question, you haven't been following along.

I wrote Adam back to let him know how it turned out. He wrote me back again, happy to hear that it worked for me. Wow, just wow.

So many things shine through in this post. In one evening I found (and patched) a bug in libpri. I learned more about Q.931 and Caller ID. I found a guy to help me put it all together. The open source development model worked. The promise of easy access to information via the internet skooled me in ISDN. Social networking proved to be very effective, even while using pre-web 2.0 technology (e-mail). Google worked (a lot).

Now I get to put it all together in this blog post to give back a little. Hopefully the next guy (or girl) trying to get some mixed up mess of SIP and ISDN devices to work together with Caller ID Name delivery will get out of the office just a little bit earlier.

AdSense

It has been three days since my first post. I feel like I have done a pretty good job letting Google/AdSense know what this blog is going to be about. I've mentioned Cisco, Asterisk, AstLinux, trixbox, BGP, etc. What gives AdSense? I'd love to specifically mention the ridiculous things you are trying to sell my readers - but I won't. I won't out of fear that you will pick up on this post and not the others, sinking me even deeper into non-relevant AdSense hell. The ads on AstLinux.org are perfect. Give me some of that!

AstLinux and trixbox

I'm convinced that people like to compare things. It doesn't matter if it's an apples to apples comparison, it doesn't matter if it's an apples to Ritz cracker comparison. Someone, somewhere will compare it if they provide even a vaguely similar function. I mean hey, if you're hungry it could come down to choosing between apples and Ritz crackers, right? They are both food, aren't they?

Yesterday someone e-mailed me and started comparing AstLinux and trixbox. This has happened before, many, many times. The difference now? I have a blog! This is where I get to finally rant about all of this.

AstLinux and trixbox are both Linux distributions. This is true. They both run Asterisk. This is true. After establishing these two facts I start to have problems finding what else they have in common.

I'm going to start this off with a disclaimer: I have nothing against trixbox. trixbox is very popular and certainly fills a need. You can't argue with that. I myself have never really used it with users, etc. but I have downloaded the ISOs and VMWare images just to check it out. I also downloaded the "source" once.

trixbox is designed to be a turn-key "do anything and everything have anything to do with Asterisk or a related application". Trixbox is based off of CentOS (a good choice based on what they are trying to do) and includes some other open source software to accomplish their goals, all integrated together.

The person that e-mailed me yesterday also asked why AstLinux was so small. After all, it is pretty much impossible to find a Compact Flash card smaller than 512MB these days. What's with this 50MB business? Why should I care to keep it small?

He then mentioned that his trixbox install was "only" 1.8GB. I had to reread it to make sure he wasn't talking about something else. Only 1.8GB? Sure enough, I look at my Trixbox 2.2 VMWare image and it comes in at about 1.3GB. I guess I never noticed. That's huge.

I very quickly realized that meant that trixbox's install size was 26 times as large as AstLinux. In my VMWare session, it also uses over 200MB of RAM (doing nothing). AstLinux comes in at about 20MB of RAM (doing nothing). That's a smaller but still significant factor of 10.

Ok, now I've done it. I am bashing trixbox. You knew it was coming, you just wanted to see how long it was going to take. You've caught me.

Maybe not. If you need something to do whatever it is trixbox does, 1.8 (or 1.3) GB isn't really that bad. It just further demonstrates, for me, how different trixbox and AstLinux are:

- different users
- different environments
- different hardware
- different goals

It was at this moment a car analogy popped into my head. What is it with people and making computer/car analogies? We've all heard about the Bill Gates car analogy (I think it might be an urban legend, though). For some reason it is perceived that people fundamentally understand cars better than they do computers and when you need to demonstrate some relationship, you can compare a computer to a car. Here's what I came up with.

Ferrari makes some of the best performance cars in the world. These things are hand built with custom selected and designed components. They offer some of the most amazing performance on the road. 0-60 in under four seconds in almost all cases. With the Enzo you can almost break 3 (0-60, 3.14 seconds). That's awesome, unless...

- You have a family of four and a two seat Italian rocket doesn't get them to school
- You can't drive stick/paddle shifters (I can't - it's a hassle)
- You can't handle 500+ horsepower (I can't)
- You don't have a Ferrari dealer near you (where will it get fixed?)

Obviously there is the other issue of cost. All of this speed, performance, and prestige will cost you at least $300,000. But because this analogy talks about software that is free (I am talking about money), we'll leave that part out (for now).

Toyota makes some great cars, trucks, and vans. They consistently outsell many other major brands. If you:

- Have a family of four (and friends)
- Can't/don't want to drive stick (it is a hassle)
- Would like a calm, quiet, smooth ride
- Have a Toyota dealer around the corner

If this is you, you probably want a Toyota Sienna. That's a minivan. It has a variety of options for entertainment (TVs, DVD players, rear seat controls). More cup holders than you can imagine, cargo space, folding seats, etc, etc. All of these features are included. Sure you might not use the DVD player everyday (some people might never use it at all) but it would really help to shut those kids up on a long car trip. C'mon, you know it does!

Back to cost. Ferrari: $300,000. Toyota: $30,000 (that's a guess). Cost, in most cases is a barrier to entry. Billionaires think nothing of it to buy a Ferrari. Why not? They've got the money!

A similar barrier to entry exists in (some) free software. My non-super-technical friends always wonder what I am doing on the computer. It took me a while to figure it out but then one day my friend Kyle just asked me "How do you use a computer without a mouse?". Then the light went on. I've been using free software, Linux, etc for so long that I think nothing of it to sit down and spend hours tinkering with config files, source code, etc. All while never touching the mouse. Now I get it.

I (along with a few other select people) think nothing of it to get an AstLinux system, open up rc.conf, tweak the network interfaces and go to town on Asterisk config files.

My friends would have nothing of it. Give them trixbox and a mouse and they will get it done their way. There is nothing wrong with that.

The end result? We both get a phone system. Mine looks more like a Ferrari (hand tuned, hand written config files, base core, etc). Their's looks more like the Sienna minivan.

In this complicated web of comparisons the difference is I am the billionaire. Only my billions are not dollars, they are hours (ok, maybe minutes or even seconds) of experience and work with software, computers, networks, etc. Billionaires (my kind or the real kind) are not everywhere. Neither are people that can make an AstLinux system work. Knowledge and experience are the barrier to entry here. Not money.

My install is faster. More efficient. Uses much less power. The hardware is cheaper. You also need to keep me around so I can make changes to it (maintenance - I am the Ferrari dealer) when it needs it.

The trixbox install gets the same basic job (phone calls) done. It can also do CRM and all of that fancy jazz if someone ever wants it. That's awesome! My super-non-technical friends provided a great service to someone and made their life/business/etc better. Bravo!

AstLinux and trixbox:
- different users
- different environments
- different hardware
- different goals

Getting Multihomed - Parts 1,2

WARNING: This is going to be a long post. You probably won't make it to the end. I guess that's what happens when I go this long without having a blog.

Since moving into our colo in Tampa way back when, Star2Star has been getting blended bandwidth from our colo facility (E-Solutions). First they had three providers (Verizon/MCI/UUnet, Global Crossing, Level3). Then two (Global Crossing, Level3).

Starting in February, Global Crossing started having some big problems. Mostly packet loss in a router in Miami. Not only did it happen frequently, it was bad (%50 - %60 loss). You can imagine what that does for VoIP...

We are obsessed with quality, so about four months ago we decided to get multihomed. Seems easy enough, right? Get the right equipment, order some circuits, do the BGP thing.

Let's start with the good equipment. We have been using the awesome Cisco Catalyst 3750 to form our redundant switch stack (two 3750G-24-TS-1U configured with STP to the colo's 4500). My buddy Anton Kapela at Five9s Data suggested them. How I love these switches:

- Good stacking (Cisco StackWise)
- Good performance (65.7mpps - that's over 65 million packets per second across the backplane)
- Good performance, with features. That's right, you can do QoS, ACLs, etc at wire speed, per port (within the limits of the TCAM, obviously).
- 24 +4 port density in 1U (24 GigE copper + four GBIC slots)
- More router-type functionality (with EMI software image - gives BGP, etc)

So with a little configuration I should be able to use one of these (right now I just grabbed a spare) to form our BGP capable router to aggregate all of these circuits. Remember those great services I talked about before? Remember the tcam? Turns out that it can only handle about 8,000 unicast routes before it starts to drop into software forwarding/otherwise start to act up. Not that big of a surprise, with the current full BGP table on the internet pushing 225K+ routes the 128MB of RAM in the 3750 wouldn't have done much good anyways. With our configuration (providers directly connected, aggregated routes only) 8000 unicast routes should be just fine. Sure we lose some end to end visibility, but it's still better than what we've got now.

It might not be the perfect equipment (it's no VXR, that's for sure) but it should get us started. Now I have to order some circuits...

We take a look at the customers we have now, the providers they have, the big providers in the area, all of that good stuff. We determine we would like to get (in no order):

- Cogent
- Verizon
- Time Warner

Cogent! Yes, I know, Cogent. Cogent sticks out on that list. Let's start with them.

Dealing with the sales guy was great. Very responsive. The price (I'm sure you know) was tough to beat. Even better than price, there was another perk...

Remember that huge global internet routing table I was talking about? There are many advertised networks in it, all with varying sizes. Some are a full Class A (/8), some are less than a Class C (/24). Or are they? It turns out that most providers/network admins/BGP snobs filter any announcement smaller than a full Class C (/24). Make sense. That table is out of control! Router memory is expensive! There is old equipment! What are those less-than-a-full-class-C small fries doing messing with BGP anyways?

What is someone with a currently small network supposed to do if they want to multihome? We need BGP to control our own routing and peer with other networks and providers. We don't have enough machines to justify the current ARIN/ISP policy of %25/%50 utilization for IP addresses to get a full class C.

It turns out that ARIN has been thinking of us. That's why there is ARIN policy 2001-2. This policy, in short, says that if you can prove you are multihoming, your ISP can give you a full Class C no questions asked. Out of the three providers mentioned above Cogent was the only provider that had even heard of this and they were more than happy to do the allocation. Thanks Cogent! (Why does everyone hate them so much?)

Don't get me wrong. I am really interested in IPv6. I know global IPv4 address space is shrinking. Hacks like NAT are running out too and it is only a matter of time before we run out of IP address space and the internet comes to a halt. Whatever. At the rate Star2Star is growing we'll need all of those IPs soon enough. When the internet really needs IPv6 all of the really smart internet god types will figure it out. I'm not worried.

So we have one provider. We have a class C. Now we need to get an ASN. Before we do that, we need to make the various contacts that ARIN requires. I started with applications for the OrgID and NocID (I think - can't remember exactly now). Much to my surprise, the turnaround time on both of these was less than one hour even though I didn't start until about 5:30 PM on a Friday. I guess they don't believe in P.O.E.T.S Day over at ARIN!

We get approved for the ASN in a day or so. Fax in the contract. Give them their $500. Now we need to wait until the good 'ol US Postal Service delivers TWO copies of our signed contract. I guess ARIN really wants to hold us to that one. Don't worry ARIN I promise we'll keep up our end of the deal. You've been great so far. Two days later (USPS Florida -> Virginia) our ASN (15092) is approved and will be in WHOIS the next day. Fantastic!

We order two more circuits. One from Time Warner, one from Verizon/MCI. Time Warner installs the circuit fairly easily. They give us a /30 and everything is good. Now we just need BGP.

So far I have filled out the Time Warner BGP request form three times. No response, not even an automated one. I have e-mailed tech support. No response, not even an automated one. I have called tech support. They say they can't do anything until I fill out the form. They say they have no requests from me. What gives?!?!? I'm giving up on them for now. At least until next week Monday. I don't give up easily.

Next we deal with Verizon. This has been interesting. Most people know of at least three Verizon-type companies:

- Verizon Wireless (cell phones)
- Verizon local (the ILEC)
- Verizon Business (used to be MCI, I guess)

So far, I have heard the following business names while trying to order/turn up this circuit:

- Verizon
- Verizon Legacy
- Verizon Core
- Verizon Business
- MCI
- UUNET

That's right: UUNET. Are you kidding me? Having six different names for your company is confusing enough. Using UUNET certainly doesn't help. Last time I heard UUNET it was the nineties and I was in middle school. I had to look it up on Wikipedia just to make sure she wasn't totally confused. Turns out it goes something like this:

UUNET -> MCI -> Verizon Business

So far their name hasn't been the only thing they are confused about. I don't even want to get into it right now. I'll make sure to update everyone as the Verizon/MCI/UUNET saga continues.

What is most surprising about all of this? Cogent. With it's horrible reputation and low cost half the people I talk to still cringe at the mention of the "C word". I can tell you this: they have been (by far) the easiest to deal with. Amazing. We'll see how the service is but as of now I am a happy Cogent customer. Anyone that would like to argue about them can try to deal with some of these other characters. Let me know how it goes!

New AstLinux Site

So I've finally put up a new AstLinux site:

http://www.astlinux.org

The wonderful web developer at Star2Star (Gabe Shepard) set me up with a super awesome Drupal install. Man these (Drupal) guys have got the web portal/blog/CMS thing down to a science. I don't have enough experience with it yet to note the details, I just know I like it. It makes me feel good. Isn't that reason enough to love it? Anyways, for now you can enjoy the new site. I plan on spending a good part of the weekend updating it - adding content, fixing outdated info, etc.