I don't "do" testimonials

I was (admittedly) getting a little bored at the office today when I realized that one of my favorite activities to pass time is checking the recent changes page of the FreeSWITCH wiki. Think about that for a minute...

If that doesn't make me a FreeSWITCH fanboy, I don't know what does. I guess I just have to finally admit it:

I am a FreeSWITCH fanboy.

How I got here I'm not really quite sure. What I do know is that I despise fanboys of all sorts (in no particular order):

- Linux
- Apple
- Microsoft (how?)
- Asterisk
- FreeSWITCH (self-loathing and denial are very powerful forces)

I've been using FreeSWITCH for over a year and it's the closest thing to a whirlwind romance I've ever had (nerdom solidified, thank you). While I'm sure the honeymoon period will end eventually at the moment I'm still smitten. I sat down and started to write a testimonial on the FreeSWITCH wiki when I realized that this was the perfect opportunity to return to blogging. I'm back and I want to talk about FreeSWITCH. Here's what I have to say (complete with an obligatory car analogy):

Star2Star has over 12,000 (business) endpoints in the field and we currently use FreeSWITCH for:

- SBC (multiple applications - customer and provider facing)
- Conference services

We're very quickly moving to IVR and (ultimately) voicemail.

What is most impressive about FreeSWITCH are the small details that make working with it an absolute joy:

- Sofia profiles (multiple unique SIP identities Just. Like. That.)
- XML flexibility (static, custom module, mod_xml_curl)
- Dialplan (conditions, PCRE, etc)
- Channel variables (access to remote media IP address and anything else in a CHANNEL VARIABLE!)
- proxy_media
- bypass_media (beautiful, just beautiful)
- RICH SIP/RTP support (SIPS, TCP, UDP, SCTP, SRTP, PAI, RPID). Yeah, I can talk to anything any way I need to.)
- Codec support. Resampling. Repacketization. It's just getting ridiculous.
- Event socket. Inbound/outbound. Async. API commands. Events.

It is still *amazing* to me that I can make FreeSWITCH into the ultimate signaling-only SBC just by adding bypass_media=true or that I can send PAI to a destination with sip_cid_type=pid. Just about every component of FreeSWITCH I've come across is very well thought out, completely logical, and completely predictable. It's like driving a German car - if you take driving seriously everything just works and everything just works exactly as you expect it to.

It's as if Anthony, Brian, Michael, and everyone else on the FreeSWITCH team are constantly sitting around, agonizing every detail of this awesome piece of software just as Hanz, Fritz, and Karl are in Munich agonizing over every detail for the next BMW 7 Series. While there are several differences in this analogy perhaps the biggest difference is that it doesn't cost $100,000 for one of the most well engineered products I've ever come across (obviously I'm not talking about the BMW).

Did I mention that in addition to this rich feature support it actually scales? All the features in the world are completely useless to me in my environment unless they can scale. We've seen again and again - FreeSWITCH scales. With OpenSER/OpenSIPS at our core and FreeSWITCH serving various edge and endpoint roles I can't identify a single point where we couldn't (ultimately) scale to meet growth or demand.

Star2Star is very unique. We have a unique product, unique approach, and unique architecture. While the architecture has generally evolved and scaled over the years with the number of users, the introduction of FreeSWITCH in a given role has always been accompanied by an excitement for what we'll be able to do for our company and our customers with our new favorite piece of software. It's a very exciting time.

The future at Star2Star is powered by our (quickly) growing customer base, FreeSWITCH, and our efforts to relentlessly deploy it where we can.

A preview: performance tests

I'm headed out the door for some sushi but I thought I'd drop in to give you an idea of what I'm working on for my next blog post. I'm hungry so let's keep this short and sweet: receive interrupt mitigation and its effects on Linux media applications.

In general I'm a big fan of receive interrupt mitigation. I'll trade some delay for a substantial decrease in system CPU time spent servicing interrupts resulting in the ability to handle more calls. I didn't just come to this one day, I've done some tests in the past to verify this.

However, I've never done a large scale test on regular, server class hardware. Usually just Asterisk on an embedded system. It usually works out well. This is why, by default, all ethernet adapters that support NAPI in Linux are enabled in AstLinux by default.

The folks at TransNexus spend a fair amount of time testing OpenSER/Kamailio/OpenSips performance. Today Jim Dalton posted the results of another test to the Kamailio User's mailing list. I replied to his post with so many questions I figured it might be time for me to lab this up myself and test my theories about interrupt handling (in Linux, specifically).

If those brighly colored rolls of fish weren't so distracting and delicious I'd promise to think about all of this over dinner. Unfortunately it will have to wait until tomorrow...

OpenSER Update

I just got a very interesting Anonymous comment to my last post. One of my BIG questions was answered:

Kamailio or OpenSIPS?

For me that question was answered - OpenSIPS. Star2Star has purchased a block of consulting time from Voice System for OpenSER support. We used it once to flush out a bug in OpenSER 1.2. Other than that we have it for a little business security. It's nice to have an official avenue for support. Plus we like (financially) supporting Open Source software. It gives us so much, we should give a little back.

Anyways, this last post makes it pretty clear: OpenSIPS is Voice System's product. That's all I needed to hear. Thanks Anonymous commenter/blog reader!

More OpenSER Drama

My last rant on this blog covered the OpenSER name change to Kamailio. It's pretty obvious how I felt about it and it's even more obvious how upset (for lack of a better word) I was with the selection of the new name.

These new developments make a name change pale in comparison. This time I have something much more serious to fret over:

There has been a fork of OpenSER.

First some history. OpenSER started life as SER. Some time ago OpenSER was forked from SER for good reason (and a common one) - the company "sponsoring" the development of SER didn't understand Open Source. Community input was ignored. Patches took forever to be applied. IPTel just didn't get it. The developers (just about all of them) set out on their own to form OpenSER and create an Open Source friendly company to serve the needs of the OpenSER community - paid support, development, consulting, etc.

Almost everyone I know has (by now) abondoned SER and moved to OpenSER. Which is good because this fork (like any other) created a certain amount of fragmentation in the community:

- documentation
- support (mailing lists, etc)
- thirdy party support
- name recognition

Even with the SER and OpenSER projects generally moving in the same direction it was clear they couldn't be treated the same. Various changes were introduced in the configuration for each piece of software. They were (and are) mostly trivial, but you can't simply move an OpenSER configuration to a SER system (or vice-versa). You also can't ask an OpenSER question on a SER list (obviously). Granted most people are subscribed to both and can provide expertise on either product but it still creates a headache for the user - fragmentation of expertise, documentation, support, etc.

As I've said this doesn't seem to be that much of a problem anymore. OpenSER (Kamailio) is clearly where it's at. It's hard for me to describe how much respect I have for this software and it's developers. It is one of the most impressive products I have ever come across. It does what it is designed to do better than just about anything else I've ever seen.

As I have come to use OpenSER more and understand it better the activist in me begins to emerge. I feel pain anytime I see someone use a product where OpenSER could clearly do the job better. OpenSER doesn't get as much credit or use as it should. I can only theorize why this is but I do know one thing:

Forking doesn't help.

Any traction (that's for you, JJ) OpenSER has made over the last few years is being seriously threatened by these political shenanigans. In the last month or so I've reviewed the first OpenSER book and solicted OpenSER contributions for the upcoming O'Reilly Asterisk Cookbook. I was starting to feel like OpenSER was finally headed towards getting the exposure it deserves.

Most developers probably don't care about these things (exposure). They should, and if they don't the project admins should. And if they don't the biz guys at the company sponsoring the development should. Here's why.

Exposure/"traction":
- Attracts more users
-- More users for testing, bug reports, etc
-- More users to write documentation
-- More users to create revenue for the sponsoring company (consulting, etc)
- Attracts more developers. Open Source development is largely ego driven and the larger and more visible the project, the more developers (both good and bad) you attract.
- Attracts more third party interest in the project. Open Source software has a lot of holes for real business use. There is a huge potential for third party projects for OpenSER. Billing systems, desktop call managers, GUIs, etc.

Neither the SER community, OpenSER community, or OpenSIPS community are large enough to sustain this fragmentation. Sure they very well might survive but they won't be what they should.

Now what? There is an OpenSER book, but no OpenSER. There is Kamailio. I hope %100 of the users figure that out. I hope the publisher of the existing OpenSER book figures out how to deal with that. What happens if everyone (well, almost everyone) bails on OpenSER/Kamailio and moves to OpenSIPS? Now anyone providing support for these other OSPs (Open Source Proxies) has to decide which they will support. SER? Kamailio? OpenSIPS? All of them? None of them? That's what worries me.

I haven't decided what I'm going to do. I still use and recommend OpenSER 1.2 so none of this affects me - yet. Of course I'll be watching this closely. I will tell you one thing... I like the name OpenSIPS more than Kamailio...

OpenSER Name Change

OpenSER has changed their name to Kamailio.

Name changes are tough. This name change is especially tough and dare I even say, personally frustrating.

Working with Star2Star is tough. We have a revolutionary architecture. We operate very differently from most other providers. Heck, we operate differently than most other companies! Don't get me wrong, this is one of the many things I like about us. That's why we're changing what people think about phone (communications) companies.

One of the main drawbacks of these differences (read: advantages) is working with legacy telco. Whether it's legacy telco people, equipment, process, or organization, I have had many, many frustrating clashes with telecom. Frustrations with technology excite me. Frustrations with people drive me crazy. People can't be fixed as easily.

This name change only exacerbates what I see as an already BIG problem. How am I going to talk to veteran telecom executives about a product who's name I can't even pronounce? Even if I do learn how to pronounce it I'll never get past it's awkwardness.

I am very proud of OpenSER. OpenSER has been crucial in making Star2Star what it is. OpenSER is amazing and I have the utmost respect for everyone involved with the project.

I am ashamed of Kamailio. Same project. Same people. Same awesome, quality software. It just doesn't feel the same. Unfortunately this is my emotional reaction to this name change. I can't help how I feel and I'm sure other people (like Alex) feel the same way.

OpenSER is a phenomenal piece of software that commands respect. It's name should too.

Maybe we'll start a new "support" mailing list. Not for people needing technical support for OpenSER/Kamailio, but support for those of us who have been personally affected by this name change.

Wow, now I'm just being dramatic!

Asterisk Cookbook

We've been trying to get the Asterisk Cookbook going. It's the same format as other O'Reilly cookbooks - practical instructions for specific problems posted as recipes, just like a cookbook you'd use in the kitchen (at least that's what people usually do with them).

I haven't contributed much but I have an excuse - I don't really work with Asterisk that much anymore. So, a couple of days ago I asked the other authors and editors if we could widen the scope of the book to include some of my favorite technologies. Anyone that reads this blog knows what that means - Cisco, OpenSER, etc. I plan on writing quite a bit about OpenSER. As I said in my OpenSER book review, OpenSER needs more press and documentation. Let's work on that!

I've now contributed three recipes! My SIP DoS/DDoS mitigation script and a couple OpenSER scripts. Some of the OpenSER stuff is a little obscure and probably won't make it to the final book. I'll be adding more OpenSER, Cisco, and SIPp stuff over the next days/weeks/months. Keep checking here and there for updates.

Oh, and if you feel like you have something to add, get a hold of one of the authors (myself, Jim, Leif, Matt, Jared, etc) and we can figure out how to get you an account on the wiki.

Book Review: Building Telephony Systems with OpenSER

I've been looking for and wondering about an OpenSER book for quite some time...

Everyone knows Asterisk is king when it comes to anything open source VoIP or telephony. OpenSER rocks but it just seems so, well, hardcore.

I can't imagine anyone walking into a Borders or Barnes and Noble, grabbing a book about OpenSER and cuddling up in one of those comfy, over-stuffed chairs on a rainy afternoon for some reading. Ok, I don't know anyone that does that anyway, and certainly not anyone with a book about OpenSER. I don't know about you but my tech books always end up getting rammed into my carry on for reading during "electronics prohibited" periods of air travel. That's all the time I care to dedicate to dead tree reads. But hey, it beats SkyMall (Jim Gaffigan: "Hey buddy, I work for SkyMall and I don't appreciate you jabbing us").

Bringing this back into focus (and on topic), is there a market for an OpenSER book? I've worked with two publishing companies in the past doing various amounts of writing and technical reviewing. Mostly O'Reilly but lately another company: Packt Publishing. Well, Packt decided there was a market for an OpenSER book and they decided to publish one. It's called Building Telephony Systems with OpenSER.

I found out about this book and knew I had to read it. What would an OpenSER book look like? What OpenSER features would be explored? What would it cost? What would it smell like? Ok, I'm just kidding about that last one but I was very interested.

I happened to be working on a couple of Packt Publishing projects at the time. I thought I'd contact one of my editors to see if they could hook me up. Turns out the book had already gone to press but they were willing to send me one. Cool!

Before I go any further I need to tell you: I won't be bought and I'm not easily influenced. Just because I've worked with Packt in the past doesn't mean I'm going to spare them any criticism or cut them any slack. If anything I'll go a little harder on them because they usually know how to get good authors and reviewers (me). ;) (Kidding, of course).

So they sent me a book. It's 295 pages and written by Flavio E. Goncalves. Flavio is a Brazilian that runs VoIP training in Brazil. Florianopolis, to be exact. Florianopolis is gorgeous. If you ever go to Brazil check it out. Hey, while you're there take an OpenSER training class from Flavio and get your company to pay for the whole thing!

Anyways, this is where my criticism starts to come in... I'm not one of these American English Nazis (does that make any sense?) that stands in line at the grocery store and freaks out when someone speaks Spanish or any other foreign language. However, I do believe that language exists as a mechanism to effectively communicate between people and sometimes that can be a little tricky even between two speakers of the same language. How many misunderstandings have you had with your friends, relatives, strangers etc while speaking the same language?

Brazilians speak Portuguese and I love it. It's one of the best sounding languages around. Being Brazilian, Flavio's first language is of course, Portuguese. It's also the language this book (Building Telephony Systems with OpenSER) was originally written in. My understanding from reading the book is that the book was developed out of Flavio's original (Portuguese) training course work and later translated to English. This makes for some very interesting and (sometimes) difficult to understand English language usage. There are several instances in the book that are difficult to follow because the language is, well, awkward. It looks like there was a proofreader... Maybe his style is different from mine but I'm pretty sure most people agree with me.

Part of me feels like this criticism is unfair and I wrestled with even mentioning it. Here's my chance to elaborate on this and explain myself.

Neither the author, editor, nor reviewers are native English language speakers yet this is what the book was written in. That's amazing to me. These people, in addition to being remarkably talented in their respective fields (publishing, technology) can all speak multiple languages and work on a team, around the world (India, Romania, Brazil) in a second language. The world is flat, my friend.

I feel like I can barely work and speak English half the time. Most Americans are the same way. We're all lazy, fat, SUV driving slobs that grunt back and forth in something that resembles English (which is already pretty "ugly"). Ok it's not that bad but that's what it's often made out to be... Anyone ever seen Idiocracy?

So for a technical book, my main criticism is English language usage and that's not even that bad. So far they're doing pretty well! On to the technical stuff...

I thought the book did a great job explaining the use of various OpenSER modules and OpenSER scripting in general. I also like how Flavio introduced readers to the various (relevant) SIP RFCs along the way. In working with OpenSER and SIP in general I can't stress how important this is. Many SIP platforms isolate you from the RFCs as much as possible. Everything is taken care of and it magically works (even with Asterisk when compared to OpenSER). With OpenSER this is not the case. You better know your RFCs. You better know the difference between components in a SIP network (Proxy, UAC, UAS, B2BUA, etc). You better know some core aspects of RFC3261 (stateful vs. stateless, strict routing, loose routing, methods, dialogs, domains, etc). Flavio isn't afraid to present this to the reader and he does it in a way that doesn't come across as snobby. Depending on the situation I've been known to quote RFCs and sections before. I probably (almost always) end up looking like a jerk but sometimes, hey, it needs to be done!

While reading I was disappointed to see so much focus on MediaProxy. I don't like MediaProxy. It's written in Python. Yuck. Enough said. I kept reading and by the end of the book Flavio admitted the mistake of more or less ignoring RTPProxy and agreed it was mostly superior. Bravo! No one knows everything, not even the guys that write the books. It was refreshing to see this in print. Once again, well done.

A couple of specific (nit picky) problems... The Asterisk config for a PSTN gateway on pg. 147 is suspect. The type doesn't make it clear which file you are working in and I doubt anyone would really use that SIP configuration in production. There's also no Zaptel config. I know this isn't an Asterisk book (there are plenty of those, the best one is free) but there should at least be a pointer to an online resource for Asterisk/Zaptel configuration or something.

The same thing goes for the Cisco config. If you look closely you'll see the dial-peer uses sip-notify for dtmf-relay. No one uses this and we certainly haven't configured OpenSER for it. Again, it's nit picky but some user could end up pulling all of his hair out trying to get this to work.

I didn't go over the OpenSER script bracket by bracket but it looked pretty good. That's even tougher to comment on because it's practically a programming language and no programmer ever completely loves the way someone else wrote something. That's just how it goes.

Overall, this is a very good book and it deserves to do well, for Flavio, everyone at Packt, OpenSER and the community in general. People who should be running OpenSER aren't. Hopefully this book can help change that!

Thank you to Flavio, the OpenSER team, and everyone at Packt Publishing!

An update on SIP DoS mitigation

After spending a couple of hours on the VoIP User's Conference last week I thought I'd keep working on my SIP DoS/DDoS script a bit and get it to the point where I'd like to run it on some of my systems (if only to collect statistics).

The new version includes several new features, the most exciting (and certainly controversial) are changes to the string pattern matching for SIP requests. I now block ALL tel: URIs by default. I don't like tel: URIs. I think they're anti-SIP and you shouldn't use them. Now my script won't let you (unless you disable it, of course).

Anyways, I had to do this because (as I've already mentioned), I changed the way pattern matching runs on SIP requests. Two big changes:

1) Support a (configurable) offset for searches into the packet.
2) Update the SIP method matches to match "$METHOD sip:"

First of all, we now (by default) only search the first 65 bytes of the packet. That should be more than enough to search the first line for the SIP method and URI. Speaking of URI, I now match the URI along with the method to prevent false matches. Before we were only matching on the method and it was causing some false positives because of things like the Allow: header (where all of the supported methods are listed).

We'll see how this goes.

One thing I wanted to bring up in the VoIP User's call last week but failed to do so is the possible use of OpenSER to protect Asterisk (and other systems) from attack. In addition to supporting cool things like SIP message length filtering (msg:len) you can also use the pike module for some basic (but slightly more intelligent) SIP rate limiting. Of course then you need to support an OpenSER config, which a lot of people don't want to do...

What else is new in the script? Basic support for udp and/or tcp, configurable bursting, fixes to the FORWARD support and more. Check it out, it's free!

More OpenSER Work

I've been working with OpenSER a lot over the last couple of weeks. I upgraded to 1.2.3, enabled TLS support, and implemented some really cool new features:

- Serial forking class
- English error messages
- Numerous code cleanups
- Much more

I'll try to find out how much of this I can share and get back to you. In the meantime I'm having a problem calling next_gw() from the LCR module. I suspect it's a db problem but I can't be sure yet...

Obscure RFC 3398 Support with OpenSER

I hate SS7. Let me say that again. I HATE SS7. I mean it's cool and everything, but there are parts of it that are unnecessarily confusing...

For instance, this week I got very confused over use of the term "cic" in telco-speak. I had thought cic was "circuit identification code". So, when one of our partners asked me to support embedding the cic in the incoming SIP INVITE, I freaked out. Why is that my responsibility? Why should my proxy ("class 5 switch") have to maintain state of individual DS0s on YOUR media gateways. This sucks! I bitterly started reading RFC 3398 (particularly section 7.2.1.1) and gave up halfway through it, becoming frustrated with the overall language of the section. After all, keeping track of DS0s just isn't my thing, and that's what CICs are about, right?

Turns out I was wrong. Keeping tracks of CICs is my thing. The problem is, these CICs are "Carrier Identification Codes", not "Circuit Identification Codes". I know everyone in networking and telecom loves to have an acronym for everything, but for GODs SAKE PEOPLE don't use the same acronym in the same context. Reading on in RFC 3398, they specifically warn for this confusion:

..

 If the 'cic=' parameter is present in the Request-URI, the
 gateway SHOULD consult local policy to make sure that it is
 appropriate to transmit this Carrier Identification Code (CIC, not to

 be confused with the MTP3 'circuit identification code') in the IAM;

..

Good point. Why didn't you say something sooner? So, now it turns out all I have to do is append a cic= (CARRIER identification code) parameter to the SIP RURI (request uniform resource identifier) on the outbound INVITE to the SIP <-> SS7 gateway. No problem. Thanks to OpenSER, module textops, and subst_uri, this is all I need to do:

avp_printf("$avp(s:cic)", "+15062"); # Set the cic value in an AVP
if (subst_uri('/^sip:([0-9]+)@(.*)$/sip:\1;cic=$avp(s:cic)@\2/i')) {
xlog("Added CIC $avp(s:cic) to RURI"); # tell us about it
};

Here is the INVITE going out to the gateway (cic in red):

U a.b.c.d:5060 -> w.x.y.z:5060
INVITE sip:14145551212;cic=+15062@w.x.y.z:5060 SIP/2.0.
Record-Route: .
Via: SIP/2.0/UDP a.b.c.d;branch=z9hG4bKf1e2.8a4d41b6.0.
Via: SIP/2.0/UDP e.f.g.h:5060;branch=z9hG4bK4e28b419;rport=5060.
From: "Polycom 320" ;tag=as477acc5f.
To: .
Contact: .
Call-ID: 343b661709e271c30bdfafe923a82adb@e.f.g.h.
CSeq: 103 INVITE.
User-Agent: Star2Star StarBox astlinux-s2s-1438-net4801.
Max-Forwards: 69.
Remote-Party-ID: "Polycom 320" ;privacy=off;screen=no.
Date: Fri, 22 Feb 2008 19:48:44 GMT.
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY.
Content-Type: application/sdp.
Content-Length: 265.
.
v=0.
o=root 6183 6184 IN IP4 e.f.g.h.
s=session.
c=IN IP4 e.f.g.h.
t=0 0.
m=audio 19770 RTP/AVP 18 0 101.
a=rtpmap:18 G729/8000.
a=fmtp:18 annexb=no.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.
a=silenceSupp:off - - - -.

Yeah, yeah, you like that?!?

I'm still not sure of the actual CIC format... That isn't EXACTLY what is shown in section 8.2.1.1 of RFC 3398 but it does look like what Cisco expects from their docs I've read (from the BTS 10200). After all, this particular SIP <-> SS7 gateway is made by them (PGW 2200).

Obviously you can get fancy with that and do all sorts of conditionals and DB calls to get the CIC, only append it on certain calls, different CICs for different sources, destinations, etc but the point is you can tweak up the RURI and insert it. I still haven't tested this against Cisco but I'm pretty sure it will work...

After I sorted out my "CIC confusion" it took me less than 30 minutes to implement this. Thanks OpenSER. You make SS7 a lot easier to deal with (as long as SIP is involved)!

Missing SIP traffic?

SIP can be cool because it resembles HTTP. OpenSER and SER are cool because they are so powerful. For instance, in OpenSER you don't need to authenticate calls. Or you can specify which request URIs should be challenged with an auth (or which method types, like REGISTER). This is usually done like so:

if (is_method("INVITE")) {

if (uri=~"^sip:1000@") {
rewritehostport("192.168.1.1:5060"); #set destination
route(1); #t_relay, etc is in route(1)
}

if (!allow_trusted()) {
if (!proxy_authorize("star2star.com","subscriber")) {
proxy_challenge("star2star.com","0");
return;
} else if (!check_from()) {
sl_send_reply("403", "Use From=ID");
return;
};

xlog("Creds are good\n");
consume_credentials();

};

if (uri=~"^sip:1[0-9]{10}@") {
route(5); #goto LCR
return;
};
};

So, in this example any SIP endpoint that can reach this proxy can hit RURI:1000 and be forwarded to the voicemail server with no authentication. As we step through this example, the only other URIs that match are after the allow_trusted or proxy_authorize checks. Basically, your source IP address has to be in the trusted table or you have to successfully respond to a 407 Proxy Authentication Required from the proxy.

I've seen this work perfectly between an Asterisk client and OpenSER hundreds of times. Most of the time it works. MOST OF THE TIME. I've noticed a scenario where it does not work, and I am struggling to figure out why...

Here's the architecture:

User's Phone -> Asterisk --(internet)--> OpenSER -> Misc. other systems

Like I said, normally this works and it looks like this (get ready for some SIP):

#
U a.b.c.d:5060 -> e.f.g.h:5060
INVITE sip:9415551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK2e8b0432;rport.
From: "User Phone" ;tag=as3474f6c4.
To: .
Contact: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 102 INVITE.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Date: Tue, 05 Feb 2008 18:24:11 GMT.
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY.
X-s2s-region: 1.
Content-Type: application/sdp.
Content-Length: 216.
.
v=0.
o=root 1429 1429 IN IP4 a.b.c.d.
s=session.
c=IN IP4 a.b.c.d.
t=0 0.
m=audio 19424 RTP/AVP 0 101.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.
a=silenceSupp:off - - - -.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 100 Trying.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK2e8b0432;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 102 INVITE.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4363 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:9415551212@e.f.g.h out_uri=sip:9415551212@e.f.g.h via_cnt==1".
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 407 Proxy Authentication Required.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK2e8b0432;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=0dd4490c85a9eb8d48ff967a8700cef0.fcb4.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 102 INVITE.
Proxy-Authenticate: Digest realm="star2star.com", nonce="valid_nonce".
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4363 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:9415551212@e.f.g.h out_uri=sip:9415551212@e.f.g.h via_cnt==1".
.

#
U a.b.c.d:5060 -> e.f.g.h:5060
ACK sip:9415551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK2e8b0432;rport.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=0dd4490c85a9eb8d48ff967a8700cef0.fcb4.
Contact: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 102 ACK.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Content-Length: 0.
.

#
U a.b.c.d:5060 -> e.f.g.h:5060
INVITE sip:9415551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport.
From: "User Phone" ;tag=as3474f6c4.
To: .
Contact: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 103 INVITE.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Proxy-Authorization: Digest username="cpesource", realm="star2star.com", algorithm=MD5, uri="sip:9415551212@e.f.g.h", nonce="valid_nonce", response="valid_response", opaque="".
Date: Tue, 05 Feb 2008 18:24:12 GMT.
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY.
X-s2s-region: 1.
Content-Type: application/sdp.
Content-Length: 216.
.
v=0.
o=root 1429 1430 IN IP4 a.b.c.d.
s=session.
c=IN IP4 a.b.c.d.
t=0 0.
m=audio 19424 RTP/AVP 0 101.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.
a=silenceSupp:off - - - -.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 100 Trying.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 103 INVITE.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4361 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:9415551212@e.f.g.h out_uri=sip:9415551212@e.f.g.h via_cnt==1".
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 100 trying -- your call is important to us.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 103 INVITE.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4361 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:9415551212@e.f.g.h out_uri=sip:18135551212@w.x.y.z:5060;transport=udp via_cnt==1".
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 183 Session Progress.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Date: Tue, 05 Feb 2008 18:24:12 GMT.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
Server: Cisco-SIPGateway/IOS-12.x.
CSeq: 103 INVITE.
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, COMET, REFER, SUBSCRIBE, NOTIFY, INFO, UPDATE, REGISTER.
Allow-Events: telephone-event.
Contact: .
Record-Route: .
Content-Disposition: session;handling=required.
Content-Type: application/sdp.
Content-Length: 238.
.
v=0.
o=CiscoSystemsSIP-GW-UserAgent 2171 1394 IN IP4 w.x.y.z.
s=SIP Call.
c=IN IP4 w.x.y.z.
t=0 0.
m=audio 18722 RTP/AVP 0 101.
c=IN IP4 w.x.y.z.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 200 OK.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK57af1f44;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Date: Tue, 05 Feb 2008 18:24:12 GMT.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
Server: Cisco-SIPGateway/IOS-12.x.
CSeq: 103 INVITE.
Allow: INVITE, OPTIONS, BYE, CANCEL, ACK, PRACK, COMET, REFER, SUBSCRIBE, NOTIFY, INFO, UPDATE, REGISTER.
Allow-Events: telephone-event.
Contact: .
Record-Route: .
Content-Type: application/sdp.
Content-Length: 238.
.
v=0.
o=CiscoSystemsSIP-GW-UserAgent 2171 1394 IN IP4 w.x.y.z.
s=SIP Call.
c=IN IP4 w.x.y.z.
t=0 0.
m=audio 18722 RTP/AVP 0 101.
c=IN IP4 w.x.y.z.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.

#
U a.b.c.d:5060 -> e.f.g.h:5060
ACK sip:18135551212@w.x.y.z:5060 SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK38cd41fe;rport.
Route: .
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Contact: .
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 103 ACK.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Content-Length: 0.
.

#
U a.b.c.d:5060 -> e.f.g.h:5060
BYE sip:18135551212@w.x.y.z:5060 SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK48cf5abc;rport.
Route: .
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
CSeq: 104 BYE.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Proxy-Authorization: Digest username="cpesource", realm="star2star.com", algorithm=MD5, uri="sip:18135551212@w.x.y.z:5060", nonce="valid_nonce", response="valid_response", opaque="".
Content-Length: 0.
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 200 OK.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK48cf5abc;rport=5060.
From: "User Phone" ;tag=as3474f6c4.
To: ;tag=F6697AC-1DCD.
Date: Tue, 05 Feb 2008 18:24:21 GMT.
Call-ID: 2ad4212700f61c3e1294439f1c72c479@a.b.c.d.
Server: Cisco-SIPGateway/IOS-12.x.
Content-Length: 0.
CSeq: 104 BYE.
.

INVITE comes in, 407 goes out and gets ACKd by remote Asterisk instance, INVITE comes back, this time with a Proxy-Authorization: header attached. The call gets forwarded, setup, and everything is fine.

HOWEVER - some times the 407 doesn't make it back to Asterisk (for whatever reason) and the call dies:

#
U a.b.c.d:5060 -> e.f.g.h:5060
INVITE sip:8135551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport.
From: "User Phone" ;tag=as664fbdbc.
To: .
Contact: .
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 INVITE.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Date: Tue, 05 Feb 2008 18:22:47 GMT.
Allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, SUBSCRIBE, NOTIFY.
X-s2s-region: 1.
Content-Type: application/sdp.
Content-Length: 216.
.
v=0.
o=root 1306 1306 IN IP4 a.b.c.d.
s=session.
c=IN IP4 a.b.c.d.
t=0 0.
m=audio 19154 RTP/AVP 0 101.
a=rtpmap:0 PCMU/8000.
a=rtpmap:101 telephone-event/8000.
a=fmtp:101 0-16.
a=silenceSupp:off - - - -.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 100 Trying.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport=5060.
From: "User Phone" ;tag=as664fbdbc.
To: .
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 INVITE.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4363 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:8135551212@e.f.g.h out_uri=sip:8135551212@e.f.g.h via_cnt==1".
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 407 Proxy Authentication Required.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport=5060.
From: "User Phone" ;tag=as664fbdbc.
To: ;tag=0dd4490c85a9eb8d48ff967a8700cef0.d2fe.
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 INVITE.
Proxy-Authenticate: Digest realm="star2star.com", nonce="valid_nonce".
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4363 req_src_ip=a.b.c.d req_src_port=5060 in_uri=sip:8135551212@e.f.g.h out_uri=sip:8135551212@e.f.g.h via_cnt==1".
.

#
U a.b.c.d:5060 -> e.f.g.h:5060
CANCEL sip:8135551212@e.f.g.h SIP/2.0.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport.
From: "User Phone" ;tag=as664fbdbc.
To: .
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 CANCEL.
User-Agent: Asterisk PBX.
Max-Forwards: 70.
Remote-Party-ID: "User Phone" ;privacy=off;screen=no.
Content-Length: 0.
.

#
U e.f.g.h:5060 -> a.b.c.d:5060
SIP/2.0 483 Too Many Hops.
Via: SIP/2.0/UDP a.b.c.d:5060;branch=z9hG4bK3beb8c07;rport=5060.
From: "User Phone" ;tag=as664fbdbc.
To: ;tag=0dd4490c85a9eb8d48ff967a8700cef0.f74a.
Call-ID: 165e2edb63fd81b30e629055245b8b28@a.b.c.d.
CSeq: 102 CANCEL.
Server: OpenSer (1.1.0-notls (i386/linux)).
Content-Length: 0.
Warning: 392 e.f.g.h:5060 "Noisy feedback tells: pid=4365 req_src_ip=e.f.g.h req_src_port=5060 in_uri=sip:8135551212@e.f.g.h out_uri=sip:8135551212@e.f.g.h via_cnt==71".
.


So, in this scenario the INVITE comes in and OpenSER responds with a 407. The remote endpoint (Asterisk) never receives the 407, gives up on the request, sending a CANCEL after about 60 seconds. At this point I'm not really sure what happens but something loops in OpenSER until Max-Forwards: is exceeded.

We can verify from packet captures on the remote Asterisk system that the 407 is not being received and therefore, Asterisk isn't resending the INVITE with auth.

What's doing on here? We're not seeing any other messages being lost, we're not seeing packet loss, what's going on?