I don't "do" testimonials

I was (admittedly) getting a little bored at the office today when I realized that one of my favorite activities to pass time is checking the recent changes page of the FreeSWITCH wiki. Think about that for a minute...

If that doesn't make me a FreeSWITCH fanboy, I don't know what does. I guess I just have to finally admit it:

I am a FreeSWITCH fanboy.

How I got here I'm not really quite sure. What I do know is that I despise fanboys of all sorts (in no particular order):

- Linux
- Apple
- Microsoft (how?)
- Asterisk
- FreeSWITCH (self-loathing and denial are very powerful forces)

I've been using FreeSWITCH for over a year and it's the closest thing to a whirlwind romance I've ever had (nerdom solidified, thank you). While I'm sure the honeymoon period will end eventually at the moment I'm still smitten. I sat down and started to write a testimonial on the FreeSWITCH wiki when I realized that this was the perfect opportunity to return to blogging. I'm back and I want to talk about FreeSWITCH. Here's what I have to say (complete with an obligatory car analogy):

Star2Star has over 12,000 (business) endpoints in the field and we currently use FreeSWITCH for:

- SBC (multiple applications - customer and provider facing)
- Conference services

We're very quickly moving to IVR and (ultimately) voicemail.

What is most impressive about FreeSWITCH are the small details that make working with it an absolute joy:

- Sofia profiles (multiple unique SIP identities Just. Like. That.)
- XML flexibility (static, custom module, mod_xml_curl)
- Dialplan (conditions, PCRE, etc)
- Channel variables (access to remote media IP address and anything else in a CHANNEL VARIABLE!)
- proxy_media
- bypass_media (beautiful, just beautiful)
- RICH SIP/RTP support (SIPS, TCP, UDP, SCTP, SRTP, PAI, RPID). Yeah, I can talk to anything any way I need to.)
- Codec support. Resampling. Repacketization. It's just getting ridiculous.
- Event socket. Inbound/outbound. Async. API commands. Events.

It is still *amazing* to me that I can make FreeSWITCH into the ultimate signaling-only SBC just by adding bypass_media=true or that I can send PAI to a destination with sip_cid_type=pid. Just about every component of FreeSWITCH I've come across is very well thought out, completely logical, and completely predictable. It's like driving a German car - if you take driving seriously everything just works and everything just works exactly as you expect it to.

It's as if Anthony, Brian, Michael, and everyone else on the FreeSWITCH team are constantly sitting around, agonizing every detail of this awesome piece of software just as Hanz, Fritz, and Karl are in Munich agonizing over every detail for the next BMW 7 Series. While there are several differences in this analogy perhaps the biggest difference is that it doesn't cost $100,000 for one of the most well engineered products I've ever come across (obviously I'm not talking about the BMW).

Did I mention that in addition to this rich feature support it actually scales? All the features in the world are completely useless to me in my environment unless they can scale. We've seen again and again - FreeSWITCH scales. With OpenSER/OpenSIPS at our core and FreeSWITCH serving various edge and endpoint roles I can't identify a single point where we couldn't (ultimately) scale to meet growth or demand.

Star2Star is very unique. We have a unique product, unique approach, and unique architecture. While the architecture has generally evolved and scaled over the years with the number of users, the introduction of FreeSWITCH in a given role has always been accompanied by an excitement for what we'll be able to do for our company and our customers with our new favorite piece of software. It's a very exciting time.

The future at Star2Star is powered by our (quickly) growing customer base, FreeSWITCH, and our efforts to relentlessly deploy it where we can.

CANCEL

I don't have a tremendous amount of time so this is going to be a short one.

There is a CANCEL related bug in Asterisk 1.4.23 versions. To be honest I'm not sure when it was introduced but I know when it was fixed:

http://bugs.digium.com/view.php?id=14431

This caused trouble for me because I often use Asterisk in tandem with OpenSIPS and FreeSWITCH. Both platforms were unable to match the CANCEL sent by Asterisk to the original INVITE. As the bug note says, this is because the CANCEL sent by Asterisk had a different branch parameter than the original INVITE. OpenSER/OpenSIPS would fail when checking t_check_trans()as long as method==CANCEL.

FreeSWITCH was a *little* easier to diagnose because it would send a 481. I suppose I could have made (and should make) my OpenSIPS configurations do this when using t_check_trans for CANCEL:

if (is_method("CANCEL")) {
if (!t_check_trans()) {
# No matching transaction, error and exit
sl_send_reply("481","Call leg/transaction does not exist");
exit;
}
# Hand it to tm
t_relay();
exit;

Anyways this has certainly been fixed in Asterisk 1.4.24. I'm looking forward to not dealing with any of this for some time...

AstLinux-FreeSWITCH ISO available for testing!

Two posts in one day. Wow.

Just as the title says and previous posts have hinted, FreeSWITCH has been added to AstLinux. Please see the official announcement on the AstLinux homepage.

AstLinux Updates

Just a few quick AstLinux updates:

1) FreeSWITCH support is just about done and has been committed to trunk. Still needs some testing but things look very promising. The installed binaries are much smaller than I thought (~3MB or so stripped and linked against uClibc). This includes some cool stuff like mod_xml_curl, mod_lua, mod_vmd, mod_snom, and more. As Tony pointed out 1.2MB of that 3MB is mod_sofia! As frustrating as that is at least mod_sofia includes IPV6, UDP, TCP, TLS, session timers, and just about every other relevant SIP standard... ;)

Small note but worth mentioning - FreeSWITCH is the first package to be committed without support for a keydisk of any kind. You need to use unionfs with this one!

2) OpenSER support has been moved to OpenSIPS. I'm still not sure about this one... I've got an existing relationship with Bogdan and all of my projects have been moved to OpenSIPS (instead of Kamailio or SER). We'll see what the community wants on this one (if AstLinux users care about a high performance SIP proxy at all).

3) RTPProxy upddate to 1.1. I love this software.

That's it for now!

The Joys of the Internet

The members of the Yahoo! Financial forum are now referring to me as "little hitler" (it's because I have a German last name, how clever) and suggesting I be sued for libel... That would get interesting. Perhaps they have never heard of The Streisand Effect?

It is unfortunate that what should be a mature, productive discussion has come to this. What is it about the internet that brings out the worst in people?

Anyways, it is very clear this issue should have never left a technical forum. I don't give stock tips; they shouldn't argue RFCs.

Yikes! (More on Sonus)

It seems that my Sonus rant has crossed over into some other arenas...

Yahoo! Financial Discussion

I've signed up for an account (kristian.kielhofner -at- yahoo.com) to participate in this thread. Should be interesting.

More on FreeSWITCH

I've hinted at it for some time:

I've been playing with FreeSWITCH.

Anyone who is reading this should already know what FreeSWITCH is and why someone (such as myself) would be so interested in it. I'm not going to go over all of that again; there are plenty of rave reviews all over the internet. I don't need to write another one (although I probably will some day).

Here's an update on what I've done so far:

1) FreeSWITCH support in AstLinux. Still coming along but much progress has already been made. It compiles cleanly (one more hack for sqlite) and appears to work. More testing soon but I was pleasantly surprised - the build system seems to be just as well designed as the rest of the project. They've done a great job!

2) I hate transcoding. Long, long ago I led an effort to re-record and convert all of the Asterisk prompts to various native file formats to avoid transcoding. More than two years later I'm doing it again for FreeSWITCH although this time I don't have to pay to re-record them all! Luckily they are made available in various sample rates already. I just had to update the script and do the converting. Big thanks to sox, Asterisk/res_convert and FreeSWITCH/Mod_native_file.

3) This one is barely worth mentioning but I've started (SVN branch, that's about it) working to re-implement recqual using FreeSWITCH to place calls. I've got some big plans for this. Let's see how much time I actually have to work on it. Don't expect much progress anytime soon.

4) Various production and consulting projects. Obviously.

As always, expect more to come!

The update you've been waiting for...

UPDATE: Any updates for this and other SIP/RTP issues can be found here.

In my last post over one month ago, I ranted on and on (big surprise, right) about some issues with Sonus equipment we were experiencing. After learning more I should elaborate on "Sonus equipment".

Like many other manufacturers Sonus has multiple products. We'll be talking about their NBS SBC. Many providers use the NBS SBC in conjunction with GSX gateways and PSX route servers. I have no comments about GSX gateways or PSX route servers; this equipment is largely transparent to us "end users". My gripes are with the NBS SBC.

Providers that use Sonus NBS:

- Level(3) (w/ GSX)
- XO (w/ PSX & GSX)
- Global Crossing
- Broadvox
- Many others

If you are using these carriers for SIP services, be aware.

Last time I was talking about timestamps. This time it's far more insidious...

Apparently (as relayed to me from Level(3) engineers) Sonus has a DSP buffer limitation for RTP packet handling. If there is ever more than a 100ms (my experience has shown it to be much less) gap in RTP Sonus will in technical terms, "freak out".

We have now identified four RTP interop issues with Sonus equipment:

1) Sonus requires all RTP packets (events or voice) to have unique timestamps. The RFCs specifically state that not only is it valid to use the same timestamp for various RTP packets, it is ideal in some cases (like events, for example).

2) The RFC 2833 events generated by Sonus equipment are goofy, to put it lightly. The event duration increments do not match the packetization of the voice stream as stated in RFC 2833 and elaborated on in RFC 4733. Specifically, Sonus equipment increments RFC 2833 duration 80 samples
at a time as if the voice stream is 10ms (regardless of what it actually is). I don't know of any other implementations that do this. Even when the audio stream is *clearly* 20 ms (in the SDP, too) Sonus will continue to increment 80 samples at a time.

3) The most recent (and biggest problem) has been caused by the Sonus (seemingly arbitrary) requirement that there never be greater than 100ms gaps in RTP. This is inherently broken behavior for robustness in IP networks.

4) Sonus has yet another issue with RTP timing and sequencing... If a call is brought up with an endpoint that clocks it's own RTP stream (IVR server, for example) everything will be fine. Until the IVR server (or whatever) bridges that channel to another device that also clocks its own RTP. Sonus (probably related to #3 above) will lose sync and drop audio for up to several seconds while it catches up to the new RTP stream. This requires those of us that work with Sonus equipment to rewrite all timestamps and sequence numbers on our equipment; which has the adverse effect of less than optimal jitter buffering (which should ideally be done at each far endpoint).

Asterisk is largely ok with all of these issues, believe it or not. The one that still causes problems is #3. If you are using Asterisk and Sonus gateways, make DAMN SURE that you are using Packet2Packet bridging and that your devices (whatever they may be) implement RFC 2833 the Sonus way. If not...

NO DTMF FOR YOU!

If you are not using Packet2Packet bridging and your events need to traverse the Asterisk core (for features, fixup, or anything else) there will be a variable length RTP gap that often exceeds the Sonus DSP buffer requirement. With gaps in RTP...

NO DTMF FOR YOU!

FreeSWITCH is also ok as long as you avoid #4. FreeSWITCH provides the configuration option to rewrite timestamps and break jitter buffering. If you are using Sonus gateways you should enable it, otherwise...

NO DTMF FOR YOU!

All of this makes me wish I was around back in the old days when there was one telco and all DTMF was inband!

Heads up!

UPDATE: Any developments on this and other SIP/RTP issues can be found here.

Some serious issues for all of those of you in SIP land:

There is a pretty serious RTP problem with Sonus equipment that has been making the rounds...

Simply put, Sonus equipment will not accept two RTP packets with the same timestamp, even if the sequence number has been properly incremented. According to various RFCs (namely 1889 and 2833) this is perfectly valid and in some cases (like video) desired.

A few slight problems... Many implementations (including Asterisk AND FreeSWITCH) will (did -more on this later) send out RFC 2833 DTMF events with the same timestamp as the last voice RTP packet. This is perfectly valid according to the RFCs mentioned above.

It appears (after my own testing) that Sonus will actually drop BOTH the voice RTP packet and the event packet. After some testing against Sonus gear it was pretty clear that no audio was being passed as long as the DTMF event occured. This makes sense because per RFC2833 a variable length DTMF event must use the same timestamp, increment the sequence counter and increase the duration when it is resent - DO NOT change the timestamp. Oh Sonus.

Both Asterisk and FreeSWITCH have incremented workarounds to address this. They are similar but there is one key difference. Asterisk now (as of SVN 12/15/2008 or so) will always use a unique timestamp for every RTP packet. I guess that solves that problem. FreeSWITCH is slightly smarter about it (as of SVN about the same time, interestingly enough) but I"m worried...

FreeSWITCH will parse the SDP to find the originator line (o=). If it is equal to "Sonus_UAC" FreeSWITCH activates a specific workaround to always send RTP packets with different timestamps. This seems more elegant but I am worried they will have to expand this hack for other equipment in the future (requiring a code change and recompile).

One could argue that Sonus has gotten this far with their current implementation and expected behavior. While it is valid (per the RFCs) to use the same timestamp, it is more /compatible/ to always use different timestamps. That appears to be what most equipment does.

This issue is what (apparantly) caused so many issues for Teliax a while back while they switched from Asterisk to FreeSWITCH. At least that's what I heard. What doesn't make any sense is that Asterisk had the same behavior as FreeSWITCH - they both sent voice and event RTP packets with identical timestamps. So that part doesn't make any sense.

Also, one would like to think that when you provide voice services (which are pretty important to your customers) you would *test* something like DTMF when you were completely switching platforms. I discovered these issues while testing Star2Star with Level(3), for example. I'm glad I was paying attention. Our customers would have been upset with broken DTMF while we updated all of our Asterisk machines (several hundred).

I'm suprised no one noticed this until mid-December or so. It will be interesting to see what other things pop out of this mess...