CANCEL

I don't have a tremendous amount of time so this is going to be a short one.

There is a CANCEL related bug in Asterisk 1.4.23 versions. To be honest I'm not sure when it was introduced but I know when it was fixed:

http://bugs.digium.com/view.php?id=14431

This caused trouble for me because I often use Asterisk in tandem with OpenSIPS and FreeSWITCH. Both platforms were unable to match the CANCEL sent by Asterisk to the original INVITE. As the bug note says, this is because the CANCEL sent by Asterisk had a different branch parameter than the original INVITE. OpenSER/OpenSIPS would fail when checking t_check_trans()as long as method==CANCEL.

FreeSWITCH was a *little* easier to diagnose because it would send a 481. I suppose I could have made (and should make) my OpenSIPS configurations do this when using t_check_trans for CANCEL:

if (is_method("CANCEL")) {
if (!t_check_trans()) {
# No matching transaction, error and exit
sl_send_reply("481","Call leg/transaction does not exist");
exit;
}
# Hand it to tm
t_relay();
exit;

Anyways this has certainly been fixed in Asterisk 1.4.24. I'm looking forward to not dealing with any of this for some time...

The update you've been waiting for...

UPDATE: Any updates for this and other SIP/RTP issues can be found here.

In my last post over one month ago, I ranted on and on (big surprise, right) about some issues with Sonus equipment we were experiencing. After learning more I should elaborate on "Sonus equipment".

Like many other manufacturers Sonus has multiple products. We'll be talking about their NBS SBC. Many providers use the NBS SBC in conjunction with GSX gateways and PSX route servers. I have no comments about GSX gateways or PSX route servers; this equipment is largely transparent to us "end users". My gripes are with the NBS SBC.

Providers that use Sonus NBS:

- Level(3) (w/ GSX)
- XO (w/ PSX & GSX)
- Global Crossing
- Broadvox
- Many others

If you are using these carriers for SIP services, be aware.

Last time I was talking about timestamps. This time it's far more insidious...

Apparently (as relayed to me from Level(3) engineers) Sonus has a DSP buffer limitation for RTP packet handling. If there is ever more than a 100ms (my experience has shown it to be much less) gap in RTP Sonus will in technical terms, "freak out".

We have now identified four RTP interop issues with Sonus equipment:

1) Sonus requires all RTP packets (events or voice) to have unique timestamps. The RFCs specifically state that not only is it valid to use the same timestamp for various RTP packets, it is ideal in some cases (like events, for example).

2) The RFC 2833 events generated by Sonus equipment are goofy, to put it lightly. The event duration increments do not match the packetization of the voice stream as stated in RFC 2833 and elaborated on in RFC 4733. Specifically, Sonus equipment increments RFC 2833 duration 80 samples
at a time as if the voice stream is 10ms (regardless of what it actually is). I don't know of any other implementations that do this. Even when the audio stream is *clearly* 20 ms (in the SDP, too) Sonus will continue to increment 80 samples at a time.

3) The most recent (and biggest problem) has been caused by the Sonus (seemingly arbitrary) requirement that there never be greater than 100ms gaps in RTP. This is inherently broken behavior for robustness in IP networks.

4) Sonus has yet another issue with RTP timing and sequencing... If a call is brought up with an endpoint that clocks it's own RTP stream (IVR server, for example) everything will be fine. Until the IVR server (or whatever) bridges that channel to another device that also clocks its own RTP. Sonus (probably related to #3 above) will lose sync and drop audio for up to several seconds while it catches up to the new RTP stream. This requires those of us that work with Sonus equipment to rewrite all timestamps and sequence numbers on our equipment; which has the adverse effect of less than optimal jitter buffering (which should ideally be done at each far endpoint).

Asterisk is largely ok with all of these issues, believe it or not. The one that still causes problems is #3. If you are using Asterisk and Sonus gateways, make DAMN SURE that you are using Packet2Packet bridging and that your devices (whatever they may be) implement RFC 2833 the Sonus way. If not...

NO DTMF FOR YOU!

If you are not using Packet2Packet bridging and your events need to traverse the Asterisk core (for features, fixup, or anything else) there will be a variable length RTP gap that often exceeds the Sonus DSP buffer requirement. With gaps in RTP...

NO DTMF FOR YOU!

FreeSWITCH is also ok as long as you avoid #4. FreeSWITCH provides the configuration option to rewrite timestamps and break jitter buffering. If you are using Sonus gateways you should enable it, otherwise...

NO DTMF FOR YOU!

All of this makes me wish I was around back in the old days when there was one telco and all DTMF was inband!

Heads up!

UPDATE: Any developments on this and other SIP/RTP issues can be found here.

Some serious issues for all of those of you in SIP land:

There is a pretty serious RTP problem with Sonus equipment that has been making the rounds...

Simply put, Sonus equipment will not accept two RTP packets with the same timestamp, even if the sequence number has been properly incremented. According to various RFCs (namely 1889 and 2833) this is perfectly valid and in some cases (like video) desired.

A few slight problems... Many implementations (including Asterisk AND FreeSWITCH) will (did -more on this later) send out RFC 2833 DTMF events with the same timestamp as the last voice RTP packet. This is perfectly valid according to the RFCs mentioned above.

It appears (after my own testing) that Sonus will actually drop BOTH the voice RTP packet and the event packet. After some testing against Sonus gear it was pretty clear that no audio was being passed as long as the DTMF event occured. This makes sense because per RFC2833 a variable length DTMF event must use the same timestamp, increment the sequence counter and increase the duration when it is resent - DO NOT change the timestamp. Oh Sonus.

Both Asterisk and FreeSWITCH have incremented workarounds to address this. They are similar but there is one key difference. Asterisk now (as of SVN 12/15/2008 or so) will always use a unique timestamp for every RTP packet. I guess that solves that problem. FreeSWITCH is slightly smarter about it (as of SVN about the same time, interestingly enough) but I"m worried...

FreeSWITCH will parse the SDP to find the originator line (o=). If it is equal to "Sonus_UAC" FreeSWITCH activates a specific workaround to always send RTP packets with different timestamps. This seems more elegant but I am worried they will have to expand this hack for other equipment in the future (requiring a code change and recompile).

One could argue that Sonus has gotten this far with their current implementation and expected behavior. While it is valid (per the RFCs) to use the same timestamp, it is more /compatible/ to always use different timestamps. That appears to be what most equipment does.

This issue is what (apparantly) caused so many issues for Teliax a while back while they switched from Asterisk to FreeSWITCH. At least that's what I heard. What doesn't make any sense is that Asterisk had the same behavior as FreeSWITCH - they both sent voice and event RTP packets with identical timestamps. So that part doesn't make any sense.

Also, one would like to think that when you provide voice services (which are pretty important to your customers) you would *test* something like DTMF when you were completely switching platforms. I discovered these issues while testing Star2Star with Level(3), for example. I'm glad I was paying attention. Our customers would have been upset with broken DTMF while we updated all of our Asterisk machines (several hundred).

I'm suprised no one noticed this until mid-December or so. It will be interesting to see what other things pop out of this mess...

Introducing Recqual

I've been waiting to talk about this one for a while.

Several months ago Star2Star was having problems with one of our upstream SIP carriers. We were starting to notice a large increase in the number of one way audio calls our customers were reporting.

When most people think of one way calls their first reaction is to blame SIP. Must be NAT! Must be a firewall! SIP sucks! Etc, etc.

I knew that wasn't the case. I just had to prove it.

I was convinced the problem wasn't SIP/UDP/IP related at all. We had multiple pcaps where we were sending RTP to the appropriate gateway. It just wasn't getting to the PSTN. Where was it going? When was this happening? Which gateways (out of hundreds) were the most problematic? We needed to know and we needed to know quickly.

I came up with and "wrote" recqual over a couple of days. After a few runs we were noticing patterns with problematic RTP endpoint IP addresses. Long story short, once these were identified we worked with the carrier to replace various bits of equipment (DSPs, line cards, etc). The one way audio problem has largely disappeared and we continue to run recqual. If this starts happening again we should know /BEFORE/ our customers do.

Of course I'm using Asterisk to place the calls. The best part of using Asterisk is it's multi-protocol flexibility. You should be able to test just about any combination of voice technologies - G.279a, G711, GSM, SIP, IAX, PRI, FXO, FXO, gtalk/jabber/jingle, skype, etc. The possibilities boggle the mind.

I've just been too busy to get it together and release this to the community - until now.

Tarball with instructions here.

Questions? Comments? Suggestions? Drop me a line.

Submit Your SIP

Ever since I've started blogging and talking about SIP people have come out of the woodwork with SIP interop problems.

After giving a talk about SIP at Astricon 2008 I received several e-mails from audience members with specific SIP issues. I LOVE getting these e-mails.

Why? I love working on SIP issues. With all of the devices using SIP there is no shortage of interop problems. Just today a guy on the Asterisk mailing list had a problem with his Cisco AS5300 and Asterisk 1.2 Usually that wouldn't be a problem at all - many people (including myself) use this combination of hardware with great success.

Why was he having problems? His AS5300 was configured for GTD and Asterisk 1.2 (apparently) doesn't handle multipart SIP bodies very well. I was able to find a patch to Asterisk 1.4 to improve multipart body parsing. That was a fun one.

I got to thinking... There should be a place where people can exchange specific SIP interop tips and notes. Otherwise how are we supposed to get anything to work!?!?

I came up with such a place and it's called SubmitYourSip.com. I' ve started to fill it in a little but hopefully (with time) it will become somewhat of a SIP wiki (with a focus on interop, of course).

I'm just getting started on it but I'll be working on my MediaWiki syntax and going back through my e-mail to dig out some of these examples.

Distro Wars!

I don't like to get into Distro Wars... Nothing is more pathetic than a bunch of FOSS geeks sitting around getting religious about:

- Distros
- Editors
- Star Wars

Ok, ok I started to embrace the stereotype a little towards the end there but if you've ever seen one of these epic battles with your own eyes you just might relate it to Star Wars too.

There are always far more serious things going on in the world and these software hippies sit around and argue about what software to edit ASCII with. Ridiculous.

However, even I will throw down when people bring up the worst idea for a distro ever:

Fedora

There, I said it. I've officially fired my first shot in a war that has been raging in the Linux community since, oh, 1991 or so.

What's wrong with Fedora, you ask? Of all of the other hundreds of distros, why would I single out Fedora and waste my time writing about it? I'll tell you why:

Much like the iPhone, it's a joke.

I CRINGE. ABSOULTELY CRINGE when I see someone trying to do something serious with Fedora. At this very minute (3am), I am typing a blog post when I should be sleeping. Why you ask? Because I just saw a post on Asterisk-Users with yet another poor soul trying to bring up Asterisk on a Fedora system (Fedora 9). I couldn't possibly sleep knowing the abomination that is Fedora+Asterisk continues.

This is a perfect example to illustrate why Fedora is such a joke. One of the fundamental principles of Linux is reliability. One of the fundamental principles of telephony is reliability. By installing Asterisk on Fedora you are flying in the face of 18 years of Linux and 100+ years of telephony.

Fedora is BLEEDING EDGE. It serves as a test bed for RedHat's next REAL Linux release. How do you feel being a tester for what is sold as a commercial product by a profitable company? Bugs in Fedora are found and fixed quickly...

For six months

You can install Fedora and deal with the usual issues in beta software. Once things finally settle down (after six months or so), you get to upgrade to the next release and start all over again!

Sounds like a great plan for a server or PBX, right?

Hosting companies sell packages based on Fedora. Shame on them. People install PBX systems with Fedora. Double shame on them. The average life span for a PBX is seven years. This means that your Fedora Asterisk system is supported with updates for %7 of the typical PBX lifespan.

CentOS/RHEL/Ubuntu LTS and several other Novell/etc offerings are supported for five years or more.

Granted Fedora has it's place. It makes a nice toy, much like the iPhone. If you want to play with cutting edge Linux, Fedora is for you and it might work on a test system, desktop, laptop, etc.

But please. Please. PLEASE do not install it on a server and don't even think of using it for Asterisk. ANYTHING else will do. Seriously, I don't have a problem with any other distro. Pick one.

Asterisk Cookbook

We've been trying to get the Asterisk Cookbook going. It's the same format as other O'Reilly cookbooks - practical instructions for specific problems posted as recipes, just like a cookbook you'd use in the kitchen (at least that's what people usually do with them).

I haven't contributed much but I have an excuse - I don't really work with Asterisk that much anymore. So, a couple of days ago I asked the other authors and editors if we could widen the scope of the book to include some of my favorite technologies. Anyone that reads this blog knows what that means - Cisco, OpenSER, etc. I plan on writing quite a bit about OpenSER. As I said in my OpenSER book review, OpenSER needs more press and documentation. Let's work on that!

I've now contributed three recipes! My SIP DoS/DDoS mitigation script and a couple OpenSER scripts. Some of the OpenSER stuff is a little obscure and probably won't make it to the final book. I'll be adding more OpenSER, Cisco, and SIPp stuff over the next days/weeks/months. Keep checking here and there for updates.

Oh, and if you feel like you have something to add, get a hold of one of the authors (myself, Jim, Leif, Matt, Jared, etc) and we can figure out how to get you an account on the wiki.

SIP DoS/DDoS Mitigation

An interesting thread came up on Asterisk-Users the other day...

Someone had problems with some miscreant attempting (and apparently succeeding) to bruteforce a SIP account on his Asterisk system.

That's a problem. Asterisk currently has no means to protect against any type of SIP flooding/bruteforce/DoS (other than running out of resources/crashing). That's ok because some people (like myself) would argue that these types of attacks are best handled via other means...

Like the kernel, or preferably the kernel of a completely seperate box.

It got me to thinking - maybe I could whip up a script using some of the cool stuff in iptables/netfilter to mitigate these SIP DoS attacks in the kernel. I don't have a ton of time to elaborate on the script but for now here it is:

http://admin.star2star.com/sipdos

It's not great. It's not very SIP-specific. Who knows how accurate/effective/resource intensive it is. It hasn't been tested (much). For all I know, it will just make things worse. However, I think it is a step in the right direction and hopefully one of many tools that can be used to protect Asterisk and other SIP/VoIP systems.

By the way, I call it "SIP DoS Mitigation" because with any large enough DDoS attack, you are toast. ..

Speaking of other tools... We're all going to celebrate America's birthday this Friday by getting together on a conference call/IRC to talk over some of these issues with the VoIP User's Conference. Hear you there?

Missing SIP traffic?

SIP can be cool because it resembles HTTP. OpenSER and SER are cool because they are so powerful. For instance, in OpenSER you don't need to authenticate calls. Or you can specify which request URIs should be challenged with an auth (or which method types, like REGISTER). This is usually done like so:

if (is_method("INVITE")) {

if (uri=~"^sip:1000@") {
rewritehostport("192.168.1.1:5060"); #set destination
route(1); #t_relay, etc is in route(1)
}

if (!allow_trusted()) {
if (!proxy_authorize("star2star.com","subscriber")) {
proxy_challenge("star2star.com","0");
return;
} else if (!check_from()) {
sl_send_reply("403", "Use From=ID");
return;
};

xlog("Creds are good\n");
consume_credentials();

};

if (uri=~"^sip:1[0-9]{10}@") {
route(5); #goto LCR
return;
};
};

So, in this example any SIP endpoint that can reach this proxy can hit RURI:1000 and be forwarded to the voicemail server with no authentication. As we step through this example, the only other URIs that match are after the allow_trusted or proxy_authorize checks. Basically, your source IP address has to be in the trusted table or you have to successfully respond to a 407 Proxy Authentication Required from the proxy.

I've seen this work perfectly between an Asterisk client and OpenSER hundreds of times. Most of the time it works. MOST OF THE TIME. I've noticed a scenario where it does not work, and I am struggling to figure out why...

Here's the architecture:

User's Phone -> Asterisk --(internet)--> OpenSER -> Misc. other systems

Like I said, normally this works and it looks like this (get ready for some SIP):

#
U a.b.c.d:5060 -> e.f.g.h:5060
INVITE sip:9415551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK2e8b0432;rport.
From: "User Phone" ;tag=as3474f6c4.
To: .
Contact: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 102 INVITE.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Date: Tue, 05 Feb 2008 18:24:11 GMT.
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY.
X-s2s-region: 1.
Content-Type: application/sdp.
Content-Length: 216.
.
v=0.
o=root 1429 1429 IN IP4 a.b.c.d.
s=session.
c=IN IP4 a.b.c.d.
t=0 0.
m=audio 19424 RTP/AVP 0 101.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.
a=silenceSupp:off - - - -.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 100 Trying.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK2e8b0432;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 102 INVITE.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4363 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:9415551212@e.f.g.h out_uri=sip:9415551212@e.f.g.h via_cnt==1".
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 407 Proxy Authentication Required.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK2e8b0432;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=0dd4490c85a9eb8d48ff967a8700cef0.fcb4.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 102 INVITE.
Proxy-Authenticate: Digest realm="star2star.com", nonce="valid_nonce".
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4363 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:9415551212@e.f.g.h out_uri=sip:9415551212@e.f.g.h via_cnt==1".
.

#
U a.b.c.d:5060 -> e.f.g.h:5060
ACK sip:9415551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK2e8b0432;rport.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=0dd4490c85a9eb8d48ff967a8700cef0.fcb4.
Contact: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 102 ACK.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Content-Length: 0.
.

#
U a.b.c.d:5060 -> e.f.g.h:5060
INVITE sip:9415551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport.
From: "User Phone" ;tag=as3474f6c4.
To: .
Contact: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 103 INVITE.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Proxy-Authorization: Digest username="cpesource", realm="star2star.com", algorithm=MD5, uri="sip:9415551212@e.f.g.h", nonce="valid_nonce", response="valid_response", opaque="".
Date: Tue, 05 Feb 2008 18:24:12 GMT.
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY.
X-s2s-region: 1.
Content-Type: application/sdp.
Content-Length: 216.
.
v=0.
o=root 1429 1430 IN IP4 a.b.c.d.
s=session.
c=IN IP4 a.b.c.d.
t=0 0.
m=audio 19424 RTP/AVP 0 101.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.
a=silenceSupp:off - - - -.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 100 Trying.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 103 INVITE.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4361 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:9415551212@e.f.g.h out_uri=sip:9415551212@e.f.g.h via_cnt==1".
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 100 trying -- your call is important to us.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 103 INVITE.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4361 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:9415551212@e.f.g.h out_uri=sip:18135551212@w.x.y.z:5060;transport=udp via_cnt==1".
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 183 Session Progress.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Date: Tue, 05 Feb 2008 18:24:12 GMT.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
Server: Cisco-SIPGateway/IOS-12.x.
CSeq: 103 INVITE.
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, COMET, REFER, SUBSCRIBE, NOTIFY, INFO, UPDATE, REGISTER.
Allow-Events: telephone-event.
Contact: .
Record-Route: .
Content-Disposition: session;handling=required.
Content-Type: application/sdp.
Content-Length: 238.
.
v=0.
o=CiscoSystemsSIP-GW-UserAgent 2171 1394 IN IP4 w.x.y.z.
s=SIP Call.
c=IN IP4 w.x.y.z.
t=0 0.
m=audio 18722 RTP/AVP 0 101.
c=IN IP4 w.x.y.z.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 200 OK.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Date: Tue, 05 Feb 2008 18:24:12 GMT.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
Server: Cisco-SIPGateway/IOS-12.x.
CSeq: 103 INVITE.
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, COMET, REFER, SUBSCRIBE, NOTIFY, INFO, UPDATE, REGISTER.
Allow-Events: telephone-event.
Contact: .
Record-Route: .
Content-Type: application/sdp.
Content-Length: 238.
.
v=0.
o=CiscoSystemsSIP-GW-UserAgent 2171 1394 IN IP4 w.x.y.z.
s=SIP Call.
c=IN IP4 w.x.y.z.
t=0 0.
m=audio 18722 RTP/AVP 0 101.
c=IN IP4 w.x.y.z.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.

#
U a.b.c.d:5060 -> e.f.g.h:5060
ACK sip:18135551212@w.x.y.z:5060 SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK38cd41fe;rport.
Route: .
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Contact: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 103 ACK.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Content-Length: 0.
.

#
U a.b.c.d:5060 -> e.f.g.h:5060
BYE sip:18135551212@w.x.y.z:5060 SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK48cf5abc;rport.
Route: .
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 104 BYE.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Proxy-Authorization: Digest username="cpesource", realm="star2star.com", algorithm=MD5, uri="sip:18135551212@w.x.y.z:5060", nonce="valid_nonce", response="valid_response", opaque="".
Content-Length: 0.
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 200 OK.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK48cf5abc;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Date: Tue, 05 Feb 2008 18:24:21 GMT.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
Server: Cisco-SIPGateway/IOS-12.x.
Content-Length: 0.
CSeq: 104 BYE.
.

INVITE comes in, 407 goes out and gets ACKd by remote Asterisk instance, INVITE comes back, this time with a Proxy-Authorization: header attached. The call gets forwarded, setup, and everything is fine.

HOWEVER - some times the 407 doesn't make it back to Asterisk (for whatever reason) and the call dies:

#
U a.b.c.d:5060 -> e.f.g.h:5060
INVITE sip:8135551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport.
From: "User Phone" ;tag=as664fbdbc.
To: .
Contact: .
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 INVITE.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Date: Tue, 05 Feb 2008 18:22:47 GMT.
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY.
X-s2s-region: 1.
Content-Type: application/sdp.
Content-Length: 216.
.
v=0.
o=root 1306 1306 IN IP4 a.b.c.d.
s=session.
c=IN IP4 a.b.c.d.
t=0 0.
m=audio 19154 RTP/AVP 0 101.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.
a=silenceSupp:off - - - -.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 100 Trying.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport=5060.
From: "User Phone" ;tag=as664fbdbc.
To: .
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 INVITE.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4363 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:8135551212@e.f.g.h out_uri=sip:8135551212@e.f.g.h via_cnt==1".
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 407 Proxy Authentication Required.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport=5060.
From: "User Phone" ;tag=as664fbdbc.
To: ;tag=0dd4490c85a9eb8d48ff967a8700cef0.d2fe.
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 INVITE.
Proxy-Authenticate: Digest realm="star2star.com", nonce="valid_nonce".
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4363 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:8135551212@e.f.g.h out_uri=sip:8135551212@e.f.g.h via_cnt==1".
.

#
U a.b.c.d:5060 -> e.f.g.h:5060
CANCEL sip:8135551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport.
From: "User Phone" ;tag=as664fbdbc.
To: .
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 CANCEL.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Content-Length: 0.
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 483 Too Many Hops.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport=5060.
From: "User Phone" ;tag=as664fbdbc.
To: ;tag=0dd4490c85a9eb8d48ff967a8700cef0.f74a.
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 CANCEL.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4365 req_src_ip=e.f.g.h req_src_port=5060 in_uri=sip:8135551212@e.f.g.h out_uri=sip:8135551212@e.f.g.h via_cnt==71".
.


So, in this scenario the INVITE comes in and OpenSER responds with a 407. The remote endpoint (Asterisk) never receives the 407, gives up on the request, sending a CANCEL after about 60 seconds. At this point I'm not really sure what happens but something loops in OpenSER until Max-Forwards: is exceeded.

We can verify from packet captures on the remote Asterisk system that the 407 is not being received and therefore, Asterisk isn't resending the INVITE with auth.

What's doing on here? We're not seeing any other messages being lost, we're not seeing packet loss, what's going on?

Finding Asterisk

I've run my own Asterisk system for quite some time. My mom lives in Minneapolis, my dad lives in Chicago and my sister lives in Montreal (stinkin' French Canadians - the worst kind). Just kidding, I love my sister and all of my Quebec friends! At this point in time I lived in Wisconsin. Cell phone plans weren't that cheap yet (especially in Canada) and my family was spread out far enough for every call to be long distance.

I started playing with Asterisk in July of 2004. It turns out that I was flying to Montreal to visit my sister during the US Independence Day Holiday (yeah, that makes sense - go to Canada for the 4th). I bought a cheap X100P clone, installed it in a Linux machine, and connected it to the analog line (SBC) at my house in Wisconsin.

I configured a basic (dangerous) Asterisk install over the course of a few days and packed my things for Montreal. Of course I also brought my laptop. I had also managed to install a softphone on it before I left. I got to my sister's apartment, connected my laptop to her VideoTron cable modem and registered with my softphone. Would it actually work? Could I make a call from this softphone to that Linux server (over the internet) and out the PSTN in Wisconsin to one of my friends - for free?

If you are reading this blog you probably know what happened next. It worked perfectly. I was amazed and my friends were impressed. This was awesome.

Over the next couple of weeks I added what I needed to my (lame) Asterisk configuration. Then disaster struck... My air conditioning broke and with it went the hard drive of the Asterisk play server. It was a play server - no backups.

I hate doing things twice. That's exactly how I felt about this. I wasn't going to work through all of that again - compiling, configuration file mess again. I swore I wasn't going to resurrect Asterisk or that machine again.

Then I heard it. Asterisk was calling. I couldn't get the idea of Asterisk out of my mind. I hit the books (voip-info.org, internet mostly) and I learned more about Asterisk - more than enough to simply rebuild my play server. A couple of days later I had it back up and better than before. In the process I also knew what I was going to do with it. I could ship an ATA to each member of my family and we could all call each other, for free. I could also get an account with an ITSP and provide cheap calling to my family members.

Today some instance of that server is still running and my family members can still call each other with a four digit dial, for free.

What's my name?

This is going to be a different kind of post. This post might actually be useful for people trying to solve this problem. Just the facts, ma'am.

One of the things that has repeatedly come up in my line of work is CallerID name delivery in PRI (Primary Rate Interface) ISDN (Integrated Services Digital Network) configurations. I learned more about CallerID name today than I ever wanted to know. Just kidding - I love getting into stuff like this!

PRI is great because call setup is fast and CallerID information is available instantly. Or is it? I always knew that Caller ID name is not carried over the PSTN (usually - in some countries it is). The number does (obviously), but the name is usually looked up in CNAM by the terminating switch, not the originating switch. What I didn't know is that sometimes this isn't done when the initial Q.931 Setup message comes down the PRI to signal a new call.

Sometimes this CNAM lookup takes a little while (fractions of a second) and the name is sent later in a separate Q.931 Facility message. This is true. Cisco says so (PDF). A Cisco ISDN-SIP gateway can be configured to do this one of two ways:

1) Wait until you receive the Q.931 Facility message with name and shove it into the SIP INVITE using either PAI (P-Asserted-Identity) or RPID (Remote-Party-ID). Send the INVITE to the SIP proxy (or wherever).

2) Send the INVITE ASAP, and then send a SIP INFO packet when the name shows up in the Q.931 Facility message.

The default is #2, which is screwy. Very cool, but still screwy. It is much harder to design a SIP platform that can accept the initial INVITE, begin to process the call, and then append the PAI or RPID information received in the later INFO.

Thanks to Cisco I now understand more about Q.931 and ISDN. Now I need to get this "thing" to work.

My test setup:

PRI -> Asterisk -> PRI -> AS5350XM -> SIP -> OpenSER -> SIP -> Device

I need to get Caller ID with name delivery through this whole mess, from the first PRI to the last SIP device.

The LEC provided the PRI coming into the Asterisk machine. I provided everything else. I saw several roadblocks:

1) Get the CID Name from the LEC (via PRI)
2) Pass it through Asterisk
3) Get it to the 5350 (via PRI)
4) Get it to OpenSER (via SIP)

Knowing what I now know about Caller ID with name in ISDN I knew just what to do for Asterisk. In zapata.conf, my incoming context is lec-in. Here it is (from extensions.conf):

[lec-in]

exten => NXXNXXXXXX,1,Wait(1)
exten => NXXNXXXXXX,n,DoSomethingElse

Yep, that's right. All you need is to Wait a little to get that second Facility IE. Asterisk doesn't support getting the Facility IE later and it certainly doesn't support sending a subsequent SIP INFO. That's a good thing because as I said the "other" way (SIP INFO) just seems goofy to me.

Now I needed to get the CallerID name to the 5350. It didn't seem to work. I start looking at "pri debug span 3" output to see the Q.931 goodness coming from Asterisk. I fired up "debug isdn q931" on the 5350. No dice. It looked like this bug in libpri was killing me:

http://bugs.digium.com/view.php?id=9651

This was committed to libpri SVN about a month ago. I update libpri from SVN, recompile Asterisk, and install the new chan_zap.so. I give it another shot. It looks like the 5350 is now getting the name over Q. 931. Using ngrep I look at the SIP INVITE coming into OpenSER from the 5350. I have an RPID header, but it looks strange. The name field in the Remote-Party-ID header is "pending". What the heck is that about? "pending" was not what I was seeing in Asterisk!

I opened up ngrep a bit to let my see any SIP INFO messages that might be coming later. Sure enough shortly after the SIP INVITE comes a SIP INFO message with my Caller ID name. Going back to my two configuration choices on the 5350 I knew I preferred option #1 (send everything in one SIP INVITE), even if it meant there was a little delay before the caller got audio. How could I configure the 5350 to wait a little and put it all in one SIP INVITE before the Cisco fired it off to OpenSER?

I dug around on cisco.com for a bit. Nothing - at least nothing obvious. You have to love Cisco configuration and Cisco docs. I decided to look around the internet and see if anyone else had this problem.

I looked on Google and found this:

http://puck.nether.net/pipermail/cisco-voip/2005-June/005485.html

I wondered if Mr. Adam Rothschild ever found the solution to his (my) problem. I open up another tab and write him an e-mail. Three minutes later (literally) he sends me this configuration snippet:

---Begin IOS Configuration---
interface Serial3/0:23
no ip address
load-interval 30
isdn switch-type primary-ni
isdn incoming-voice modem
isdn supp-service name calling
isdn negotiate-bchan
no isdn outgoing display-ie
no cdp enable
exit
gateway
timer receive-rtp 1200
sip-ua
disable-early-media 180
retry invite 3
retry response 3
retry bye 3
retry cancel 3
timers buffer-invite 500

---End IOS Configuration---

Let's get away from this technical mumbo-jumbo and talk about people for a minute...

Mr. Adam Rothschild got an e-mail from a random stranger across the internet referencing an obscure technical problem that he had over two years ago. In less than three minutes he dug up the solution and wrote me back. I have a SmartNet support contract on this 5350 but I doubt the techs at Cisco could have helped me any better or faster than a nice guy (Adam) helping a stranger (me).

Wipe away your tears, you sentimental fool. We're getting back to configuration. This blog is hardcore. Couldn't you tell?

I applied Adam's config to my AS5350XM running IOS 12.4(15)T. Here is the SIP INVITE from the 5350 to OpenSER:

U 192.168.0.1:61306 -> 192.168.0.10:5060
INVITE sip:9418675309@192.168.0.10:5060 SIP/2.0.
Via: SIP/2.0/UDP 192.168.0.1:5060;x-ds0num="ISDN 3/1:D 3/1:DS1
1:DS0";branch=z9hG4bK901AB9.
Remote-Party-ID: "STAR2STAR COMM"
;party=calling;screen=no;privacy=off.
From: "STAR2STAR COMM" ;tag=971D8C-1203.
To: .
Date: Mon, 16 Jul 2007 21:57:24 GMT.
Call-ID: 5E99C7DC-331E11DC-8126E6C7-399CBB13@192.168.0.1.
Supported: 100rel,timer,resource-priority,replaces.
Min-SE: 1800.
Cisco-Guid: 1586976572-857608668-2150694933-1673067056.
User-Agent: Cisco-SIPGateway/IOS-12.x.
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, UPDATE, REFER,
SUBSCRIBE, NOTIFY, INFO, REGISTER.
CSeq: 101 INVITE.
Max-Forwards: 70.
Timestamp: 1184623044.
Contact: .
Expires: 300.
Allow-Events: telephone-event.
Content-Type: application/sdp.
Content-Disposition: session;handling=required.
Content-Length: 288.
.
v=0.
o=CiscoSystemsSIP-GW-UserAgent 7275 8957 IN IP4 192.168.0.1.
s=SIP Call.
c=IN IP4 192.168.0.1.
t=0 0.
m=audio 20746 RTP/AVP 18 0 101.
c=IN IP4 192.168.0.1.
a=rtpmap:18 G729/8000.
a=fmtp:18 annexb=no.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16

Yeah yeah! Look at that Caller ID name in that Remote-Party-ID header! I feel like that's the best looking SIP INVITE I have ever seen. How does one SIP INVITE look better than any other? If you don't know the answer to that question, you haven't been following along.

I wrote Adam back to let him know how it turned out. He wrote me back again, happy to hear that it worked for me. Wow, just wow.

So many things shine through in this post. In one evening I found (and patched) a bug in libpri. I learned more about Q.931 and Caller ID. I found a guy to help me put it all together. The open source development model worked. The promise of easy access to information via the internet skooled me in ISDN. Social networking proved to be very effective, even while using pre-web 2.0 technology (e-mail). Google worked (a lot).

Now I get to put it all together in this blog post to give back a little. Hopefully the next guy (or girl) trying to get some mixed up mess of SIP and ISDN devices to work together with Caller ID Name delivery will get out of the office just a little bit earlier.