A ClueCon Preview...

A while back I saw a preview for the new A-Team movie. While the movie itself looks horrible I was reminded of the original TV series with its many interesting characters and catch phrases. Among my personal favorites?

I love it when a plan comes together.

That's exactly how I feel with one of my "pet projects" from the past couple of months. Much like Hanibel and the A-Team I was up against formidable issues in trying to accomplish my task: implementing a flexible (very flexible), reasonably high performance LCR server that could be added to my existing architecture.

First I needed to select an LCR "engine". Multiple possibilities were considered but I left the final recommendation up to the DB and billing teams I work with. They selected mod_lcr from FreeSWITCH. While I was certain droute from OpenSIPS (or something similar) would have higher performance I accepted their recommendation. After playing with mod_lcr a bit I can also see its potential.

So now the question was: can FreeSWITCH respond with the proper SIP signaling (300 Multiple Choices)? Using the redirect application from mod_dptools it could not. I created a bounty to add multiple Contact/300 Multiple Choices functionality to FreeSWITCH. Tony had it implemented that day.

With the ability to respond properly I now had to get the data. Mod_lcr looked nice but it certainly wasn't designed for this application. All of the default syntax, tables, etc showed it being used with FreeSWITCH for FreeSWITCH. The tables and code used several bridge specific syntax examples. I hacked mod_lcr to return data to mod_dptools/redirect properly. A created a JIRA issue with my patch and a couple of days later Rupa had it committed.

So now FreeSWITCH could be a route server. All I needed to do was make sure OpenSIPS could route from what FreeSWITCH returned. Turns out it could not. RFC 3261 (section 21.3.1) states "...the SIP response MAY contain several Contact fields or a list of addresses in a Contact field." The Sofia stack from FreeSWITCH used multiple Contact headers, each with its own URI. OpenSIPS would only parse the first one returned. Sofia couldn't be changed easily so OpenSIPS would need to be changed (it was non-compliant anyway). Without this change there is no ability to handle multiple contacts and only the first would be used. It could be worse but obviously this wasn't good enough.

I contacted Bogdan from OpenSIPS to see what it would take to update the parser to handle multiple Contact headers. He indicated it would take four hours or so. Once he got back to me I had an OpenSIPS system that would handle multiple contact headers and create new branches from a failure route as desired.

So how did it all turn out? Well, you have two ways to hear the end of this story:

1) Attend ClueCon at the Trump Hotel in Chicago, IL in early August.
2) Wait until mid-August for an update here.

I'll make sure to post all of my materials - conference presentation, sipp scenarios for testing, OpenSIPS configuration, FreeSWITCH configuration, DB tweaks, etc.

Too late to make it to ClueCon this year? Just make sure to register next year, I'm sure I'll be there.

Upcoming Review: Building Telephony Systems with OpenSIPS 1.6

Packt Publishing has once again asked me to review their latest work in the OpenSIPS series: Building Telephony Systems with OpenSIPS 1.6.

My review of the previous edition goes all the way back to when OpenSIPS was called OpenSER. I have another post discussing that topic...

Anyways, I should be receiving the book this week and I should have a review up by next week.

I don't "do" testimonials

I was (admittedly) getting a little bored at the office today when I realized that one of my favorite activities to pass time is checking the recent changes page of the FreeSWITCH wiki. Think about that for a minute...

If that doesn't make me a FreeSWITCH fanboy, I don't know what does. I guess I just have to finally admit it:

I am a FreeSWITCH fanboy.

How I got here I'm not really quite sure. What I do know is that I despise fanboys of all sorts (in no particular order):

- Linux
- Apple
- Microsoft (how?)
- Asterisk
- FreeSWITCH (self-loathing and denial are very powerful forces)

I've been using FreeSWITCH for over a year and it's the closest thing to a whirlwind romance I've ever had (nerdom solidified, thank you). While I'm sure the honeymoon period will end eventually at the moment I'm still smitten. I sat down and started to write a testimonial on the FreeSWITCH wiki when I realized that this was the perfect opportunity to return to blogging. I'm back and I want to talk about FreeSWITCH. Here's what I have to say (complete with an obligatory car analogy):

Star2Star has over 12,000 (business) endpoints in the field and we currently use FreeSWITCH for:

- SBC (multiple applications - customer and provider facing)
- Conference services

We're very quickly moving to IVR and (ultimately) voicemail.

What is most impressive about FreeSWITCH are the small details that make working with it an absolute joy:

- Sofia profiles (multiple unique SIP identities Just. Like. That.)
- XML flexibility (static, custom module, mod_xml_curl)
- Dialplan (conditions, PCRE, etc)
- Channel variables (access to remote media IP address and anything else in a CHANNEL VARIABLE!)
- proxy_media
- bypass_media (beautiful, just beautiful)
- RICH SIP/RTP support (SIPS, TCP, UDP, SCTP, SRTP, PAI, RPID). Yeah, I can talk to anything any way I need to.)
- Codec support. Resampling. Repacketization. It's just getting ridiculous.
- Event socket. Inbound/outbound. Async. API commands. Events.

It is still *amazing* to me that I can make FreeSWITCH into the ultimate signaling-only SBC just by adding bypass_media=true or that I can send PAI to a destination with sip_cid_type=pid. Just about every component of FreeSWITCH I've come across is very well thought out, completely logical, and completely predictable. It's like driving a German car - if you take driving seriously everything just works and everything just works exactly as you expect it to.

It's as if Anthony, Brian, Michael, and everyone else on the FreeSWITCH team are constantly sitting around, agonizing every detail of this awesome piece of software just as Hanz, Fritz, and Karl are in Munich agonizing over every detail for the next BMW 7 Series. While there are several differences in this analogy perhaps the biggest difference is that it doesn't cost $100,000 for one of the most well engineered products I've ever come across (obviously I'm not talking about the BMW).

Did I mention that in addition to this rich feature support it actually scales? All the features in the world are completely useless to me in my environment unless they can scale. We've seen again and again - FreeSWITCH scales. With OpenSER/OpenSIPS at our core and FreeSWITCH serving various edge and endpoint roles I can't identify a single point where we couldn't (ultimately) scale to meet growth or demand.

Star2Star is very unique. We have a unique product, unique approach, and unique architecture. While the architecture has generally evolved and scaled over the years with the number of users, the introduction of FreeSWITCH in a given role has always been accompanied by an excitement for what we'll be able to do for our company and our customers with our new favorite piece of software. It's a very exciting time.

The future at Star2Star is powered by our (quickly) growing customer base, FreeSWITCH, and our efforts to relentlessly deploy it where we can.

CANCEL

I don't have a tremendous amount of time so this is going to be a short one.

There is a CANCEL related bug in Asterisk 1.4.23 versions. To be honest I'm not sure when it was introduced but I know when it was fixed:

http://bugs.digium.com/view.php?id=14431

This caused trouble for me because I often use Asterisk in tandem with OpenSIPS and FreeSWITCH. Both platforms were unable to match the CANCEL sent by Asterisk to the original INVITE. As the bug note says, this is because the CANCEL sent by Asterisk had a different branch parameter than the original INVITE. OpenSER/OpenSIPS would fail when checking t_check_trans()as long as method==CANCEL.

FreeSWITCH was a *little* easier to diagnose because it would send a 481. I suppose I could have made (and should make) my OpenSIPS configurations do this when using t_check_trans for CANCEL:

if (is_method("CANCEL")) {
if (!t_check_trans()) {
# No matching transaction, error and exit
sl_send_reply("481","Call leg/transaction does not exist");
exit;
}
# Hand it to tm
t_relay();
exit;

Anyways this has certainly been fixed in Asterisk 1.4.24. I'm looking forward to not dealing with any of this for some time...

AstLinux Updates

Just a few quick AstLinux updates:

1) FreeSWITCH support is just about done and has been committed to trunk. Still needs some testing but things look very promising. The installed binaries are much smaller than I thought (~3MB or so stripped and linked against uClibc). This includes some cool stuff like mod_xml_curl, mod_lua, mod_vmd, mod_snom, and more. As Tony pointed out 1.2MB of that 3MB is mod_sofia! As frustrating as that is at least mod_sofia includes IPV6, UDP, TCP, TLS, session timers, and just about every other relevant SIP standard... ;)

Small note but worth mentioning - FreeSWITCH is the first package to be committed without support for a keydisk of any kind. You need to use unionfs with this one!

2) OpenSER support has been moved to OpenSIPS. I'm still not sure about this one... I've got an existing relationship with Bogdan and all of my projects have been moved to OpenSIPS (instead of Kamailio or SER). We'll see what the community wants on this one (if AstLinux users care about a high performance SIP proxy at all).

3) RTPProxy upddate to 1.1. I love this software.

That's it for now!

A preview: performance tests

I'm headed out the door for some sushi but I thought I'd drop in to give you an idea of what I'm working on for my next blog post. I'm hungry so let's keep this short and sweet: receive interrupt mitigation and its effects on Linux media applications.

In general I'm a big fan of receive interrupt mitigation. I'll trade some delay for a substantial decrease in system CPU time spent servicing interrupts resulting in the ability to handle more calls. I didn't just come to this one day, I've done some tests in the past to verify this.

However, I've never done a large scale test on regular, server class hardware. Usually just Asterisk on an embedded system. It usually works out well. This is why, by default, all ethernet adapters that support NAPI in Linux are enabled in AstLinux by default.

The folks at TransNexus spend a fair amount of time testing OpenSER/Kamailio/OpenSips performance. Today Jim Dalton posted the results of another test to the Kamailio User's mailing list. I replied to his post with so many questions I figured it might be time for me to lab this up myself and test my theories about interrupt handling (in Linux, specifically).

If those brighly colored rolls of fish weren't so distracting and delicious I'd promise to think about all of this over dinner. Unfortunately it will have to wait until tomorrow...

OpenSER Update

I just got a very interesting Anonymous comment to my last post. One of my BIG questions was answered:

Kamailio or OpenSIPS?

For me that question was answered - OpenSIPS. Star2Star has purchased a block of consulting time from Voice System for OpenSER support. We used it once to flush out a bug in OpenSER 1.2. Other than that we have it for a little business security. It's nice to have an official avenue for support. Plus we like (financially) supporting Open Source software. It gives us so much, we should give a little back.

Anyways, this last post makes it pretty clear: OpenSIPS is Voice System's product. That's all I needed to hear. Thanks Anonymous commenter/blog reader!

More OpenSER Drama

My last rant on this blog covered the OpenSER name change to Kamailio. It's pretty obvious how I felt about it and it's even more obvious how upset (for lack of a better word) I was with the selection of the new name.

These new developments make a name change pale in comparison. This time I have something much more serious to fret over:

There has been a fork of OpenSER.

First some history. OpenSER started life as SER. Some time ago OpenSER was forked from SER for good reason (and a common one) - the company "sponsoring" the development of SER didn't understand Open Source. Community input was ignored. Patches took forever to be applied. IPTel just didn't get it. The developers (just about all of them) set out on their own to form OpenSER and create an Open Source friendly company to serve the needs of the OpenSER community - paid support, development, consulting, etc.

Almost everyone I know has (by now) abondoned SER and moved to OpenSER. Which is good because this fork (like any other) created a certain amount of fragmentation in the community:

- documentation
- support (mailing lists, etc)
- thirdy party support
- name recognition

Even with the SER and OpenSER projects generally moving in the same direction it was clear they couldn't be treated the same. Various changes were introduced in the configuration for each piece of software. They were (and are) mostly trivial, but you can't simply move an OpenSER configuration to a SER system (or vice-versa). You also can't ask an OpenSER question on a SER list (obviously). Granted most people are subscribed to both and can provide expertise on either product but it still creates a headache for the user - fragmentation of expertise, documentation, support, etc.

As I've said this doesn't seem to be that much of a problem anymore. OpenSER (Kamailio) is clearly where it's at. It's hard for me to describe how much respect I have for this software and it's developers. It is one of the most impressive products I have ever come across. It does what it is designed to do better than just about anything else I've ever seen.

As I have come to use OpenSER more and understand it better the activist in me begins to emerge. I feel pain anytime I see someone use a product where OpenSER could clearly do the job better. OpenSER doesn't get as much credit or use as it should. I can only theorize why this is but I do know one thing:

Forking doesn't help.

Any traction (that's for you, JJ) OpenSER has made over the last few years is being seriously threatened by these political shenanigans. In the last month or so I've reviewed the first OpenSER book and solicted OpenSER contributions for the upcoming O'Reilly Asterisk Cookbook. I was starting to feel like OpenSER was finally headed towards getting the exposure it deserves.

Most developers probably don't care about these things (exposure). They should, and if they don't the project admins should. And if they don't the biz guys at the company sponsoring the development should. Here's why.

Exposure/"traction":
- Attracts more users
-- More users for testing, bug reports, etc
-- More users to write documentation
-- More users to create revenue for the sponsoring company (consulting, etc)
- Attracts more developers. Open Source development is largely ego driven and the larger and more visible the project, the more developers (both good and bad) you attract.
- Attracts more third party interest in the project. Open Source software has a lot of holes for real business use. There is a huge potential for third party projects for OpenSER. Billing systems, desktop call managers, GUIs, etc.

Neither the SER community, OpenSER community, or OpenSIPS community are large enough to sustain this fragmentation. Sure they very well might survive but they won't be what they should.

Now what? There is an OpenSER book, but no OpenSER. There is Kamailio. I hope %100 of the users figure that out. I hope the publisher of the existing OpenSER book figures out how to deal with that. What happens if everyone (well, almost everyone) bails on OpenSER/Kamailio and moves to OpenSIPS? Now anyone providing support for these other OSPs (Open Source Proxies) has to decide which they will support. SER? Kamailio? OpenSIPS? All of them? None of them? That's what worries me.

I haven't decided what I'm going to do. I still use and recommend OpenSER 1.2 so none of this affects me - yet. Of course I'll be watching this closely. I will tell you one thing... I like the name OpenSIPS more than Kamailio...