tag:blogger.com,1999:blog-92209328111428930572024-03-13T06:51:38.671-04:00Not Just AstLinux StuffI created AstLinux but I write and rant about a lot of other things here. Mostly rants about SIP and the other various technologies I deal with on a daily basis.Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.comBlogger77125tag:blogger.com,1999:blog-9220932811142893057.post-25232700593247391602014-06-23T10:07:00.000-04:002014-08-15T12:35:07.656-04:00Verto - WebRTC and FreeSWITCH Get HitchedUnless you've been hiding under a rock you know that WebRTC is posed to be the next big thing in real time communications.<br />
<br />
If you're familiar with the technical details of WebRTC you also know that WebRTC doesn't mandate a signaling protocol - that's left up to well, whoever. For many of us coming from a SIP/telephony background it's made the most sense to use a signaling protocol we all know - SIP.<br />
<br />
WebRTC makes extensive use of WebSockets and this combined with various JavaScript SIP libraries (<a href="http://sipml5.org/" target="_blank">sipml5</a>, <a href="http://sipjs.com/" target="_blank">sipjs</a>, <a href="http://jssip.net/" target="_blank">jssip</a>, etc) allows you to do call control using SIP from a browser to a remote system over WebSockets. This assumes, of course, that the remote system supports <a href="http://www.rfc-editor.org/rfc/rfc7118.txt" target="_blank">SIP over WebSocket transport</a>. Then there's also the matter of implementing the various requirements for WebRTC media support such as ICE, STUN, DTLS, SRTP, etc. In short, implementing full WebRTC support is no small task.<br />
<br />
Over a year ago I worked with the FreeSWITCH developers to get support for this approach to WebRTC added to FreeSWITCH. As usual, Tony and team delivered a very impressive result - the ability to use the aforementioned JavaScript libraries to call into a FreeSWITCH system using audio, video, or both - all from your browser! Of course once you were connected to a FreeSWITCH system all of the existing functionality was available to you. Conferencing, bridging to existing/legacy endpoints, etc. It wasn't a stretch at all to connect from Chrome using ICE, SRTP, OPUS, and SIP over secure WebSocket and then bridge to an endpoint using SIP over UDP and G729 (or even a PRI)! It boggles the mind to consider what is happening to the audio alone in this scenario - encrypting/decrypting, transcoding, and resampling. All magically and masterfully handled by FreeSWITCH.<br />
<br />
One of the most interesting aspects of WebRTC is the ability to develop applications using the full power of the web and the browser environment. For years now we've seen WebRTC sample applications emerge demonstrating just how easy and powerful the peer-to-peer capabilities of WebRTC are. Some of these even demonstrate increasingly complicated "multi-peer" scenarios involving multiple parties.<br />
<br />
Where does FreeSWITCH fit with all of these new-fangled web technologies? First, I've already mentioned the ability for FreeSWITCH to bridge to endpoints on different networks. Even though WebRTC shares some standards with these legacy endpoints, direct communication or even media will almost certainly never be possible. Requiring DTLS and SRTP virtually guarantees that.<br />
<br />
FreeSWITCH also hosts many powerful applications - including voicemail, conferencing, and even entire custom applications written using LUA or Javascript. Custom applications can also be developed using <a href="https://wiki.freeswitch.org/wiki/Mod_event_socket" target="_blank">ESL</a> - a powerful event socket with <a href="https://wiki.freeswitch.org/wiki/Event_Socket_Library" target="_blank">accompanying library</a>.<br />
<br />
Historically there hasn't been a good way to make use of all of this functionality - the power of HTML5/JS/CSS in the browser (via WebRTC) and the power of voice/video applications hosted in the cloud using FreeSWITCH.<br />
<br />
Until now.<br />
<br />
Verto is an exciting new Javascript library and FreeSWITCH endpoint module. Together they allow web developers to use a single Javascript library for call control and FreeSWITCH event handling and interaction. That's right - call control and ESL in the same endpoint/protocol/library!<br />
<br />
The FreeSWITCH endpoint module is configured to listen on a WebSocket or Secure WebSocket (or both). The Javascript library is included and configured to point to the FreeSWITCH instance. With a single library and simple API a web developer can make full use of a remote FreeSWITCH system using WebRTC within minutes!<br />
<br />
One more thing - WebRTC (by way of OPUS) supports two channel audio (stereo). As part of Verto development the FreeSWITCH team decided it was finally time to implement stereo as well!<br />
<br />
Make sure to stay tuned for more about WebRTC and Verto but in the meantime - <a href="https://webrtc.freeswitch.org/verto/" target="_blank">enjoy the demo</a>!Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-13566918584659249802014-05-02T15:31:00.000-04:002014-05-02T15:31:09.019-04:00VoIP Users Conference Talks CryptoI spent about two hours today on the always awesome <a href="http://www.voipusersconference.org/">VoIP Users Conference</a> talking with Olle Johansson, Dan York, Tim Panton, Dave Taht, and others about VoIP security and encryption in general.<br />
<br />
If you can handle listening to my voice and seeing my stupid mug it should be pretty educational. I hope you appreciate the Nagel print hanging on my office wall!<br />
<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="405" src="//www.youtube.com/embed/S1jGz0u7_-4" width="540"></iframe>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com2tag:blogger.com,1999:blog-9220932811142893057.post-3098730386516571032014-03-26T12:12:00.002-04:002014-03-26T19:07:48.316-04:002013 Kamailio Awards<span style="font-family: Arial, Helvetica, sans-serif;">It has come to my attention that Daniel-Constantin Mierla has selected my blog as a <a href="http://by-miconda.blogspot.com/2014/03/kamailio-2013-awards.html">2013 Kamailio Award Winner</a>! This is a huge honor and I appreciate the recognition from Daniel and the entire Kamailio project.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">With only six posts in all of 2013 it seems that (thankfully for me) some people do prefer quality over quantity.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Thanks again Daniel!</span>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-54226917643571405072014-03-11T09:51:00.000-04:002014-03-26T19:07:36.826-04:00Securing Real-time Communications - Quickly<div dir="ltr" id="docs-internal-guid-0b91363b-b167-e308-75c4-5efb6e0ec589" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">This is a post that would be received completely differently a year ago.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">We now live in a “post-Snowden” era where even average members of the general public are at least vaguely aware of things like “wiretaps” and “encryption”. With that said this isn’t another post about those revelations.</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Over the past year I’ve been steadily working to improve the security of the various real-time communications platforms I’m involved in. Timing on this has been coincidental with the news about the NSA and the various programs operated by them. Of course with these revelations I’ve been paying extra special attention to key sizes, cipher suites, and of course the use of elliptic curve cryptography.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">As regular readers of my blog might suspect my primary technical focus is on SIP and RTP. SIP provides for the use of TLS to secure the signaling channel. While TLS isn’t always TLS (TLS versions, cipher suites, etc) there isn’t anything special about the application of TLS to SIP.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">TLS is one of those things we just take for granted these days (kids - get off my lawn)! However, there are </span><a href="http://en.wikipedia.org/wiki/Transport_Layer_Security#Attacks_against_TLS.2FSSL" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;" target=""><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">several well known issues</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> with various TLS implementations currently in use. Even though TLS is used to secure banking and highly sensitive web traffic everyday, SSL Labs reports that only </span><a href="https://www.trustworthyinternet.org/ssl-pulse/" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">20% of web servers</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> currently support TLS 1.2. Yikes.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Of course TLS isn’t just for web traffic. SIP can be secured with TLS as well. This is a very interesting scenario because as the name implies, SIP is almost exclusively used as a signaling channel to establish another session (which may or may not be secured). This means that securing SIP with TLS is not only important for the integrity of the SIP session, it’s often reused to secure the communications established by that session. This is the case with real time voice and video communications using RTP or more specifically the secure profile of RTP (SRTP).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">As always, anything SIP related is complicated and security is no exception. To offer a completely encrypted solution the alphabet soup and resulting complexity adds up pretty quickly. First, you must establish a secure signaling channel using SIP over TLS. You then use SDES (hopefully) to exchange offers and key information over SDP. Once the SDP offer answer model has done it’s thing you will have negotiated the transport of secure voice and/or video using SRTP.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">Originally defined in </span><a href="http://tools.ietf.org/html/rfc3711" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">RFC 3711</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">, SRTP has now been updated with several follow-ups, errata, and extensions. Looking around in the open source community and commercial VoIP space, however, shows that many implementations have not moved very far beyond this original specification for various reasons.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">One of these reasons is due to existing libraries implementing SRTP. Just about all of the open source implementations of SRTP that I’m aware of use some version of the </span><a href="http://srtp.sourceforge.net/srtp.html" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">original libsrtp</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> written by David McGrew of Cisco Systems over seven years ago. Since then libsrtp has been forked, patched, and otherwise used by pjsip, FreeSWITCH, Asterisk, the Chromium/Chrome web browsers from Google (for WebRTC), and countless other projects.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">For some time now I’ve been interested in replacing the built in AES and SHA functions in libsrtp with those provided by OpenSSL. This goal led me to the work of John Foley (also of Cisco Systems) and the official libsrtp Github repository. When I first started looking into this John had (recently) completed work on a </span><a href="https://github.com/cisco/libsrtp/tree/feature-openssl" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">branch of libsrtp</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> that accomplishes exactly this. He also went two steps further: the use of AES-NI and support for AES-GCM mode in SRTP.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">What?</span><br />
<br />
<span style="color: #1155cc; font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: underline; vertical-align: baseline;"><a href="http://en.wikipedia.org/wiki/AES_instruction_set" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;">AES-NI</a></span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> is the reason I was first interested in using OpenSSL for SRTP. Long story short new(ish) Intel CPUs include a new instruction set that greatly increases the performance of AES (especially with AES-GCM and larger payload sizes). This is a good thing and should (hopefully) lead to more widespread adoption of SRTP.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">The use of OpenSSL included another perk: support for </span><a href="http://en.wikipedia.org/wiki/Galois/Counter_Mode" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">AES-GCM</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">. While AES-GCM isn’t necessarily new there hadn’t been support for it with SRTP yet. AES-GCM is the current “gold standard” AES block cipher mode of operation, especially for “streaming” operations like VPNs and other packetized data. In fact, </span><a href="http://www.cisco.com/web/about/security/intelligence/nextgen_crypto.html" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">Cisco currently considers</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> AES-GCM a “next generation encryption” solution. AES-GCM is also preferred by the National Security Agency’s (NSA) </span><a href="http://www.nsa.gov/ia/programs/suiteb_cryptography/" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">Suite B crypto specification</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> (for better or worse in a “post-Snowden” world).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">AES-GCM is one of those “standing on the shoulders of giants” moments I really appreciate. To think that I’m writing a blog post in 2014 talking about a technology that uses a </span><a href="http://en.wikipedia.org/wiki/Galois_field" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">field</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> named after an early nineteenth century </span><a href="http://en.wikipedia.org/wiki/%C3%89variste_Galois" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">French mathematician</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> (with a really interesting life) is amazing to me. As Isaac Newton said “If I have seen further it is by standing on the shoulders of giants.” In this case the giant was born over 200 years ago and only lived to the age of 20.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">Anyway, John Foley (another giant) was able to provide a </span><a href="https://github.com/jfigus/PJSIP" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">fork of pjsip</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> that included the feature-openssl branch of libsrtp and had been patched to support AES-GCM mode with SIP (SDP/SDES) and SRTP. He just didn’t have anything else to test it against.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">I wrote patches implementing 128 bit AES-GCM mode using the updated libsrtp branch for both </span><a href="http://jira.freeswitch.org/browse/FS-5937" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">FreeSWITCH</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> and </span><a href="https://issues.asterisk.org/jira/browse/ASTERISK-22832" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">Asterisk</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">. Because of limitations in the original SRTP specification both projects would require fairly significant refactoring to include support for 192 and 256 bit GCM modes, “</span><a href="http://tools.ietf.org/html/rfc6188" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">big AES SRTP</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">”, and the use of GCM mode with DTLS (for WebRTC). In fact, both FreeSWITCH and Asterisk implemented support for SRTP by largely differentiating only on authentication tag length.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">In any case we were able to use these patches to test 128 bit AES-GCM mode SRTP interoperability between pjsip, Asterisk, and FreeSWITCH. However, there was still work to be done.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">I’m happy to report that as of last week the FreeSWITCH master branch now not only has support for the use of 128, 192 and 256 bit AES-GCM modes it also includes a </span><a href="http://jira.freeswitch.org/browse/FS-5755" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">very flexible</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> means to control the use of various SRTP crypto suites on both incoming and outgoing channels. Unfortunately this hasn’t been implemented for DTLS use with WebRTC yet but I plan to advocate for that shortly. I’m also still working with the Asterisk project to clean up my patch for inclusion there.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">I think it’s safe to say that with these changes FreeSWITCH now supports state of the art (some may say “next generation”) cryptography with SRTP. However, if you recall the keys used for SRTP are still exchanged using SIP over TLS, which depending on various factors, could still have numerous issues and be fundamentally insecure. How can we be sure that our SIP signaling channel is properly secured?</span><br />
<br />
<span style="color: #1155cc; font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: underline; vertical-align: baseline;"><a href="https://jira.freeswitch.org/browse/FS-5839" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;">FreeSWITCH</a></span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> and </span><a href="http://kamailio.org/docs/modules/4.1.x/modules/tls.html#tls.p.cipher_list" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">Kamailio</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> now both also support the explicit specification of TLS version and cipher suite. When using OpenSSL 1.0.1 (which is required by FreeSWITCH) this allows the user to require the use of TLS 1.2 and only the strongest cipher suites supported by OpenSSL. Of course this is great news for FreeSWITCH and Kamailio, but what about the rest of the SIP+SRTP implementations out there?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Unfortunately support for these standards is bleeding edge at this point and not widely deployed. While any use of SIP over TLS and SRTP is certainly preferred for various reasons (performance and security) I’m pushing for support of TLS 1.2 and AES-GCM everywhere possible.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Hopefully I’ll be able to offer more on that soon but for now enjoy the faster and more secure real time communications made possible using FreeSWITCH and Kamailio!</span>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com3tag:blogger.com,1999:blog-9220932811142893057.post-40157385342972309072013-12-10T15:24:00.001-05:002014-03-26T19:08:29.747-04:00High Quality Entropy<span style="font-family: Arial, Helvetica, sans-serif;"><b id="docs-internal-guid-75030b4c-de2c-447d-08c7-f000974a7751" style="font-weight: normal;"></b><br />
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<b id="docs-internal-guid-75030b4c-de2c-447d-08c7-f000974a7751" style="font-weight: normal;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is a (relatively) short post that I’ve been meaning to write for a while. A recent </span><a href="http://arstechnica.com/security/2013/12/we-cannot-trust-intel-and-vias-chip-based-crypto-freebsd-developers-say/" style="text-decoration: none;" target="_blank"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Ars Technica story</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (and the resulting </span><a href="https://news.ycombinator.com/item?id=6880474" style="text-decoration: none;" target="_blank"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">HN discussion</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) prompted me to finally sit down and write this.</span></span></b></div>
<span style="line-height: 1.15;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: black; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span></span>
<span id="docs-internal-guid-75030b4c-de2c-447d-08c7-f000974a7751" style="line-height: 1.15;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: black; vertical-align: baseline; white-space: pre-wrap;">I’ve been aware of software defined radios for quite some time. The entire concept seemed very cool (especially because I’ve always been interested in radio). Unfortunately they were also somewhat </span><a href="https://www.ettus.com/product/details/UB100D-BDL" style="text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">expensive</span></a><span style="color: black; vertical-align: baseline; white-space: pre-wrap;">.</span></span></span><br />
<span style="line-height: 1.15;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: black; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span></span>
<span id="docs-internal-guid-75030b4c-de2c-447d-08c7-f000974a7751" style="line-height: 1.15;"><span style="font-family: Arial, Helvetica, sans-serif;"><span style="color: black; vertical-align: baseline; white-space: pre-wrap;">The </span><a href="http://sdr.osmocom.org/trac/wiki/rtl-sdr" style="text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">RTL-SDR series</span></a><span style="color: black; vertical-align: baseline; white-space: pre-wrap;"> of </span><a href="http://www.amazon.com/RTL-SDR-RTL2832U-Popular-Software-Packages/dp/B00C37AZXK/ref=sr_1_1?ie=UTF8&qid=1386704327&sr=8-1&keywords=rtl+sdr" style="text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">dongles</span></a><span style="color: black; vertical-align: baseline; white-space: pre-wrap;"> changed all of that. For less than $20 you could enter the exciting world of software defined radio - using </span><a href="http://gnuradio.org/redmine/projects/gnuradio/wiki" style="text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">GNURadio</span></a><span style="color: black; vertical-align: baseline; white-space: pre-wrap;"> and other programs to apply open source and programming to radio. It’s all very cool stuff but I struggled to find a practical application. Then I found rtl_entropy.</span></span></span><br />
<span style="line-height: 1.15;"><span style="color: black; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline; white-space: pre-wrap;"><br /></span></span>
<span id="docs-internal-guid-75030b4c-de2c-447d-08c7-f000974a7751" style="line-height: 1.15;"><span style="color: black; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline; white-space: pre-wrap;">This time last year you would never hear words like “encryption”, “cryptography”, and “entropy” on news broadcasts like CNN. Then Edward Snowden starting revealing the reach of the National Security Agency’s various programs and since then we’ve been bombarded with a new revelation on an almost weekly basis.</span></span><br />
<b id="docs-internal-guid-75030b4c-de2c-447d-08c7-f000974a7751" style="font-weight: normal;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial, Helvetica, sans-serif; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ideas that would have seemed crazy and paranoid just a year ago now seem remarkably likely. Backdoors in closed source programs. Submarines tapping undersea cables. Data collection on a massive scale. Tampering of hardware by various state agencies (the NSA and counterparts in China, for example).</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Just today Ars Technica posted a story about the FreeBSD developers no longer explicitly trusting hardware RNG modules from Intel (</span><a href="http://en.wikipedia.org/wiki/RdRand" style="text-decoration: none;" target="_blank"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">RDRAND</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">) and Via (</span><a href="http://www.via.com.tw/en/initiatives/padlock/hardware.jsp" style="text-decoration: none;" target="_blank"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">padlock</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">). Note that I say explicitly because unlike Linux (which has always mixed all available sources of entropy) FreeBSD uses one hardware source explicitly. With all of the questions surrounding the (potential) involvement of the NSA in these designs this probably wasn’t the best idea. In FreeBSD 10 that’s going to change.</span></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial, Helvetica, sans-serif; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">However, RDRAND and padlock were always very good (supposedly) sources for LOTS of entropy. With their use being called into question, what are we to do for applications that require large amounts of available system entropy but are (hopefully) free from tampering and yet readily available?</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">That’s where </span><a href="https://github.com/pwarren/rtl-entropy" style="text-decoration: none;" target="_blank"><span style="background-color: transparent; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">rtl_entropy</span></a><span style="background-color: transparent; color: black; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> comes in. I first found this project back in August. rtl_entropy uses atmospheric radio noise from an RTL-SDR dongle to generate entropy. Because the RTL-SDR hardware is able to sample about 3.2 MS/S this translates into about </span><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3.4Mbits/s of entropy. That’s a lot, especially when you consider what some high price (yet still proprietary and unknown) hardware entropy sources </span><a href="http://en.wikipedia.org/wiki/Comparison_of_hardware_random_number_generators" style="text-decoration: none;" target="_blank"><span style="background-color: white; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">cost</span></a><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">rtl_entropy is open source and available for inspection. It runs with cheap, mass-produced radio dongles which are (presumably) difficult to “cook” for this purpose. As the author states “it samples atmospheric noise, does Von-Neumann debiasing, runs it through the FIPS 140-2 tests, optionally (-e) does Kaminsky debiasing if it passes the FIPS tests, then writes to the output. It can be run as a daemon which by default writes to a FIFO, which can be read by rngd to add entropy to the system pool”. All of this for less than $20.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When compared to hardware random number generator devices from the Wikipedia table we can see just how revolutionary this is. With proprietary hardware (and in some cases software), which is still unavailable for inspection, the next cheapest source of high quality entropy is $235 per Mbit/s. With a $15 RTL-SDR dongle rtl_entropy is about $6 per Mbit/s and the entire software stack is open source. As said before, it’s also ready to use by just about any Linux system via rngd.</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Also, because Linux always mixes the sources of available entropy rtl_entropy can be safely combined with various other mechanisms available - RDRAND, </span><a href="http://www.issihosts.com/haveged/" style="text-decoration: none;" target="_blank"><span style="background-color: white; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">haveged</span></a><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, </span><a href="http://www.vanheusden.com/aed/" style="text-decoration: none;" target="_blank"><span style="background-color: white; color: #1155cc; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">aed</span></a><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">, etc.</span></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: white; color: #222222; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></span><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I’m proud my company and I have been able to sponsor Paul and his work on rtl_entropy. This is what open source is all about!</span></div>
</b>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-52649015885561819452013-09-23T10:51:00.000-04:002014-03-26T19:09:11.060-04:00Learning "Stupid NAT Tricks" from Apple<span style="font-family: Arial, Helvetica, sans-serif;">In my <a href="http://blog.krisk.org/2013/09/apples-new-facetime-sip-perspective.html">last post</a> I spent some time (and many words) describing the what, how, and why behind some of the changes Apple has made to the Facetime protocol over the past three years.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
Needless to say I was impressed. There were several things those of us in the open source world can learn from that analysis:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">- SIP, STUN, RTP, RTCP port multiplexing (while n</span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">ot including the a=rtcp-mux </span><a href="http://tools.ietf.org/html/draft-ietf-avt-rtp-and-rtcp-mux-07#section-5.1.1" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;" target="_blank">attribute</a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">)</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- Compact SIP headers</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- SDP minimization (removing rtpmap lines, etc)</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- SDP compression</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"><span style="line-height: 1.15;">The IETF is currently working on port multiplexing (mainly via WebRTC) but I have yet to see a proposal that includes multiplexing signaling and media on the same port (with the </span><span style="line-height: 18px;">architecture</span><span style="line-height: 1.15;"> of WebRTC that's not a surprise). Either way I'll consider this "in progress". I guess there's always </span><a href="http://en.wikipedia.org/wiki/Inter-Asterisk_eXchange" style="line-height: 1.15;" target="_blank">IAX</a><span style="line-height: 1.15;"> ;).</span></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Compact SIP headers are already supported by a variety of open source applications. We'll consider this one done.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">SDP minimization is something that can't generally be done in a standards compliant way. You'll note I labelled these optimizations as a "slight protocol violation" in my previous post. We'll table this one for now.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"><span style="line-height: 1.15;">SDP compression, however, is very interesting. SIP bares more than a </span><span style="line-height: 17px;">resemblance</span><span style="line-height: 1.15;"> to HTTP and <a href="http://en.wikipedia.org/wiki/HTTP_compression" target="_blank">HTTP compression</a> has been standard and well supported for quite some time. With that in mind I see no reason why it can't become more widely supported for SIP applications as well. It's also important to point out that while I'm calling this "SDP compression" it's really SIP body compression that could be used for any MIME type or combinations of MIME type (multipart).</span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">As we saw from my last post this is no easy task. The parsing of plaintext SDPs is so easy and commonplace today that anything else is a significant undertaking. For compressed SIP bodies to be transparent to me in my everyday life they would need to be supported in at least the following projects/products:</span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">- <a href="http://www.freeswitch.org/" target="_blank">FreeSWITCH</a></span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">- <a href="http://www.kamailio.org/w/" target="_blank">Kamailio</a></span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">- <a href="http://www.pjsip.org/" target="_blank">PJSIP</a></span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">- <a href="http://www.wireshark.org/" target="_blank">Wireshark</a></span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">- <a href="http://www.sipcapture.org/" target="_blank">HOMER</a></span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">- <a href="http://www.ixiacom.com/products/network_test/applications/ixload/index.php" target="_blank">Ixia IXLoad</a> (probably not going to happen)</span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;"><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><span style="line-height: 1.15;">Fortunately I have good news! Shortly after my last post </span></span>Daniel-Constantin Mierla from Kamailio contacted me and let me know that he had written the <a href="http://git.sip-router.org/cgi-bin/gitweb.cgi/sip-router/?a=commit;h=bd716593efb83d4124c0c0cb824bf158d547ca0d" target="_blank">gzcompress module</a> for "the fun of a quick coding in the evening". There's still work to be done on this module but this was an awesome (and unexpected) start!</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;">In the same day I was also contacted by ashdnazg from the <a href="https://code.google.com/p/pyreshark/" target="_blank">Pyreshark</a> project. First of all, for a protocol junky like myself the ability to write Wireshark protocol dissectors on the fly in Python is very, very exciting. I can't believe I hadn't heard of this project before! Anyway, within hours of receiving the Facetime packet capture ashdnazg was able to provide me with a Pyreshark-based Wireshark dissector (in 26 lines of Python, no less)! My local (Ubuntu) instance of Wireshark now looks like this when loading a Facetime packet capture:</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-VL02BQRlfiE/Uj3QijWOCVI/AAAAAAAAAVQ/gSZ9nMKoyvg/s1600/pyreshark.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Arial, Helvetica, sans-serif;"><img border="0" src="http://2.bp.blogspot.com/-VL02BQRlfiE/Uj3QijWOCVI/AAAAAAAAAVQ/gSZ9nMKoyvg/s640/pyreshark.png" height="496" width="640" /></span></a></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;">Oh yeah!</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;">This is another marvel of open source - within 24 hours of my last post the leaders of two open source projects from across the globe were able to collaborate with me to support the functionality I described here. Amazing.</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;">Of course we're not done yet. I created a <a href="http://jira.freeswitch.org/browse/FS-5814" target="_blank">bounty</a> for FreeSWITCH to support this functionality as well. Anthony and the boys are very busy but we'll see if I can get any traction on this one.</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;">That leaves PJSIP and HOMER. Seeing as HOMER is based on Kamailio it's probably not that difficult to implement this. I'd also like to think (based on the use case and project goals) that they may be more interested in this than anyone.</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;">Stay tuned!</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<br /></div>
Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com2tag:blogger.com,1999:blog-9220932811142893057.post-41857081776465384902013-09-20T08:42:00.001-04:002014-03-26T19:13:03.897-04:00Apple's new Facetime - a SIP Perspective<div dir="ltr" id="docs-internal-guid--7734b3f-3956-b633-583d-d8e27edb78b5" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">A colleague of mine recently sent over a PCAP file containing an Apple Facetime session between two iPhone devices running just-released iOS 7. Being a <a href="http://blog.krisk.org/2013/02/packets-of-death.html">protocol junkie</a> he thought I might be interested in seeing them. Clearly that’s the case!</span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">As always (because I’m lazy) I looked around the internet to see what other research people had done on the “Facetime protocol”. I found several excellent (though now somewhat dated) </span><a href="http://blog.roychowdhury.org/2010/06/25/facetime-on-iphone-4-vanilla-unencrypted-stun-and-sip/" style="line-height: 1.15; text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">articles</span></a><span style="vertical-align: baseline;">. Without giving it away just yet it looks like Apple has been very busy over the last three years (like you needed me to tell you that). As we usually do here, let’s start looking at packets!</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">At first glance Facetime 2013 resembles any normal SIP capture:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
<span style="vertical-align: baseline;"></span>
</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-T4BezMWd4E4/Uju8sv5mAMI/AAAAAAAAAUk/wVoPRtNFmqU/s1600/capture.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Arial, Helvetica, sans-serif;"><img border="0" src="http://2.bp.blogspot.com/-T4BezMWd4E4/Uju8sv5mAMI/AAAAAAAAAUk/wVoPRtNFmqU/s640/capture.png" height="302" width="640" /></span></a></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
<span style="vertical-align: baseline;"></span>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Then some interesting details begin to emerge… What are these unknown packets? What’s up with this INVITE?</span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Let’s talk about the unknown UDP packets. We’ll select one and force Wireshark to decode it as RTP:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
<span style="vertical-align: baseline;"></span>
</span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-mQPSZnzrL3o/Uju83XFKOVI/AAAAAAAAAUs/vWLP22nIvlY/s1600/decode-as-rtp.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Arial, Helvetica, sans-serif;"><img border="0" src="http://2.bp.blogspot.com/-mQPSZnzrL3o/Uju83XFKOVI/AAAAAAAAAUs/vWLP22nIvlY/s640/decode-as-rtp.png" height="310" width="640" /></span></a></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
<span style="vertical-align: baseline;"></span>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">That worked and we can see this is RTP using a dynamic payload type. Our RTP packets are now correctly dissected but Wireshark seems to have confused the SIP and STUN packets for RTP. Why?</span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Forcing Wireshark to decode a given stream (where “stream” is a SRC IP:PORT pair and a DST IP:PORT pair) attempts to force a decode of that protocol type on all packets belonging to that stream. So what happened here?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">If we look closer at the UDP layer we can see that STUN, SIP, and RTP all appear to be using the same port number on each endpoint (16402 in this case). We also have no idea what codec payload type 104 is (we’d need to see the SDP for that). Now’s probably a good time to look at the SIP signaling a bit closer.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
<span style="vertical-align: baseline;"></span>
</span><br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-oGEfaD70zds/Uju-FU88jfI/AAAAAAAAAVA/mbvIm0QVzUU/s1600/facetime-invite.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: Arial, Helvetica, sans-serif;"><img border="0" src="http://2.bp.blogspot.com/-oGEfaD70zds/Uju-FU88jfI/AAAAAAAAAVA/mbvIm0QVzUU/s640/facetime-invite.png" height="289" width="640" /></span></a></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
<span style="vertical-align: baseline;"></span>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">Apple is using compact SIP headers (interesting). The User-Agent uses the codename for Facetime and GK more than likely implies the use of Apple’s </span><a href="https://developer.apple.com/library/ios/documentation/GameKit/Reference/GKVoiceChatService_Class/Reference/Reference.html#//apple_ref/doc/uid/TP40008261-CH1-DontLinkElementID_1" style="text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">GameKit</span></a><span style="vertical-align: baseline;">. That SDP, however, looks a little strange. Certainly not the simple, printable ASCII we’re used to seeing in SIP bodies!</span></span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">However, not all hope is lost. SIP compact header “e” maps to encoding. Encoding “deflate” is pretty standard HTTP 1.1 compression (from RFC 1951). Wireshark clearly doesn’t support this with SIP so we’ll have to do a little more work…</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">I saved the SIP message bodies (SDPs) from the INVITE and 200 OK into separate files. Here’s what hexdump had to say about the INVITE:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />
<span style="vertical-align: baseline;"></span>
</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000000 da78 4f75 6e41 3083 bc10 f123 3f07 4da0</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000010 36bc e021 0f95 2886 d46d 4224 8a26 057a</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000020 554a 20d4 9010 bfbc a6eb 7352 d641 3d6a</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000030 3b33 f5e3 734d ebdf cbf4 b9db aa6b fd3a</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000040 a62a 1ebc 746e 9c65 eece 76c8 c059 1620</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000050 080b 85a3 b090 8800 6f7c 6dd4 3657 da97</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000060 2af7 373d 6a53 2b93 39c1 4fe5 c29a af7c</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000070 dbd0 8e7d 6b67 c714 bdc3 6a59 00a0 28dd</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000080 721e 0cf5 3fb8 4c59 624d 4c52 92ad dfb8</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000090 323a ee23 0555 ac68 960a 49f2 032e b77c</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">00000a0 22e8 8737 bac4 9a9e f7cc 5d5a 3f5c 8e9a</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">00000b0 1841 c170 29ec 9a5b c673 d380 7c7a 1545</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">00000c0 98b2 057e b082 5410 9ce0 54c3 eaf5 e1d7</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">00000d0 67d0 f53b 98ca e594 db45 ea5f ab31 e487</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">00000e0 55d2 2cdf f888 ba7d 6ddf 8494 8e28 5ae5</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">00000f0 d2c4 c571 8555 75ab effc 2f77 e58e e580</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000100 f3a3 594f 2acd 1b21 0aeb 5467 97da 07d4</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000110 770c 03fc 5b01 3c74 </span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">0000118</span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Oh, you can’t read that? Yeah, I can’t either. We’ll have to inflate these. Unfortunately no standard utility (gunzip, etc) seemed to want to uncompress these binary blobs so I had to hack up some PHP (first thing I could find with Google, whatever):</span><br />
<span style="font-family: Times, 'Times New Roman', serif; line-height: 1.15;"><br /></span>
<span style="font-family: Times, 'Times New Roman', serif; line-height: 1.15;">#!/usr/bin/php</span><br />
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<div style="line-height: 1.15;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">//Add the usual php open and close statements (thanks Blogger)</span></div>
<span style="font-family: Times, Times New Roman, serif; line-height: 17px; vertical-align: baseline;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;"></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">$filename = "./my-sdp";</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">$handle = fopen($filename, "rb");</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">$contents = fread($handle, filesize($filename));</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">fclose($handle);</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">$uncompressed = gzuncompress($contents);</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<div style="line-height: 1.15;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">echo $uncompressed;</span></div>
</div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Which gave me a legible SDP offer from the INVITE:</span><br />
<span style="font-family: Times, 'Times New Roman', serif; line-height: 1.15;"><br /></span>
<span style="font-family: Times, 'Times New Roman', serif; line-height: 1.15;">v=0</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">o=GKVoiceChatService 0 0 IN IP4 192.168.231.118</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">s=mobile</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">c=IN IP4 192.168.231.118</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">b=AS:2000</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">t=0 0</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=FLS;VRA:0;MVRA:0;RVRA1:1;AS:2;MS:-1;LTR;CABAC;CR:3;LF:-1;PR;CH:4;AR:4/3,3/4;XR;</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=DMBR</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=CAP</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">m=audio 16402 RTP/AVP 104 105 106 9 0 124 122 121</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=rtcp:16402</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:AAC SamplesPerBlock 480</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=rtpID:3189937293</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=au:65792</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:104 sbr;block 480</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:105 sbr;block 480</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:106 sec;sbr;block 480</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:122 sec</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:121 sec</span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">Now we can begin to analyze what’s actually happening here. Our INVITE contains a perfectly valid audio offer advertising support for PCMU (payload type 0), 16kHz G722 (payload 9), and six dynamic payload types. The IANA has defined </span><a href="http://www.iana.org/assignments/rtp-parameters/rtp-parameters.xhtml#rtp-parameters-1" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">static payload types</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> for RTP. Simply put, we know payload type 0 is PCMU but anything between 96-127 SHOULD (</span><a href="http://tools.ietf.org/html/rfc4566" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">RFC 4566</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">) have a corresponding rtpmap line to map RTP payload type to a “media encoding name”. Looking at this trace alone I don’t know what these RTP payload types are.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">However, it’s very likely that the payload types have stayed the same even though Apple has now removed the suggested rtpmap lines. More than likely Apple has “hardcoded” these payload type codec maps internally. We’ll get to why in just a bit.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">The other iPhone 5 running iOS 7 responds with the following SDP answer in the 200 OK:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="line-height: 1.15;"><span style="font-family: Times, Times New Roman, serif;">v=0</span></span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">o=GKVoiceChatService 0 0 IN IP4 192.168.231.100</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">s=mobile</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">c=IN IP4 192.168.231.100</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">b=AS:2000:2000</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">t=0 0</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=FLS;VRA:0;MVRA:0;RVRA1:1;AS:2;MS:-1;LTR;CABAC;CR:3;LF:-1;PR;CH:4;AR:4/3,3/4;XR;</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=DMBR</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=CAP</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">m=audio 16402 RTP/AVP 104 106 121 122</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=rtcp:16402</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:AAC SamplesPerBlock 480</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=rtpID:3770747611</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=au:65792</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:104 sbr;block 480</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:106 sec;sbr;block 480</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:121 sec</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Times, Times New Roman, serif; vertical-align: baseline;">a=fmtp:122 sec</span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Using what we know from the 2010 analysis (complete with rtpmap lines) it seems we have agreed to use the AAC_ELD codec at 24kHz and 16kHz sample rates. However, absent analysis of an RTP stream with payloads 121 or 122 (and missing rtpmap lines) it’s hard for me to say what those other payload types represent.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">A casual reader may have read this far and thought to themselves: at the end of the day we’re ending up with 24kHz AAC_ELD audio just like we were in 2010. Apple has gone through all of this work for nothing. Oh and by the way, how/why are SIP, STUN, RTP, and RTCP using the same UDP port?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">It’s called </span><a href="http://tools.ietf.org/html/draft-ietf-avt-rtp-and-rtcp-mux-07" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;" target="_blank">port multiplexing</a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"> and it has become all the rage these days (it’s standard in WebRTC, for example - although not to this extent). Unfortunately I’m not able to determine if port multiplexing is new to Facetime 2013 (I’m sure someone will chime in here). Either way it’s an important technical distinction. Typically a SIP session of this type would require at least three UDP ports:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">- SIP signaling</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- RTP for audio</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">- RTCP for well, </span><a href="http://en.wikipedia.org/wiki/RTP_Control_Protocol" style="text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">RTCP</span></a><span style="vertical-align: baseline;"></span></span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">What’s wrong with three ports? Three ports make it much more difficult (if not impossible) to cross some types of NAT devices and firewalls. Three ports and three possibly bad interactions with some firewall or other device. Three times as much exposure to </span><a href="http://en.wikipedia.org/wiki/Murphy's_law" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;" target="_blank">Murphy's law</a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">. With port multiplexing Apple has greatly increased the chances that Facetime will work through challenging network environments. With this aggressive (I’ve never seen it before) use of UDP multiplexing for all of these standard protocols Apple has virtually guaranteed that if the signalling works the media and everything else will too.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17px;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="line-height: 17px;">UPDATE: The always on-point </span><a href="https://twitter.com/oej" style="line-height: 17px;" target="_blank">Olle Johansson</a><span style="line-height: 17px;"> has postulated that Apple might be multiplexing everything over a single port for greater </span><a href="http://www.rfc-editor.org/rfc/rfc7021.txt" style="line-height: 17px;" target="_blank">compatibility</a><span style="line-height: 17px;"> with carrier grade NAT (CGN) implementations, which place specific limits on the number of ports used per client. Thanks Olle!</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Any SIP engineer will tell you NAT is the bane of our existence. That’s why (even from 2010) Apple has made use of protocols and features such as STUN, ICE, TURN, etc. These are discussed all over the internet so I won’t get into them here but in summary they are all technologies used to traverse NAT devices and network firewalls.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">It is very clear Apple has made Facetime (finally) ready for primetime. Let me explain why and how I came to that conclusion. First let’s talk strategy.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">Without going into all of the details it is my opinion that Skype became as popular as it is because (like many successes) “it just works”. The main reason Skype “just works” is its almost-magical NAT traversal. The creators of Skype learned a lot from their previous gig defeating firewalls for peer to peer music sharing with Kazaa. Both Skype and Kazaa have an almost legendary reputation for NAT and firewall traversal. If there is a way through a NAT device or firewall they will probably figure it out. It’s with this technology (and timing, codecs, etc) that Skype became so popular. When nothing else worked, Skype would (and still does). It just works (and as we’ve seen that’s </span><a href="http://about.skype.com/press/2011/05/microsoft_to_acquire_skype.html" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">hugely</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> valuable).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">The more endpoints that “just work” the more </span><a href="http://en.wikipedia.org/wiki/Metcalfe%27s_law" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">Metcalfe’s law</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> comes into effect. If Apple can succeed in making as many endpoints as possible “just work” they have a hugely valuable real-time communications network with Facetime.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">A skeptic might point out that Apple is already using STUN, TURN, and ICE. They’ve already got NAT “figured out”. Generally speaking, yes. However, they’ve now taken some extraordinary steps to take their NAT handling (and by extension the “value” of Facetime) to the next level. Apple wants Facetime to work on as many networks as possible and they’ve spent a lot of time making sure of that. Let’s look at the changes from Facetime of 2010 to now:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">- SIP, STUN, RTP, RTCP port multiplexing (while n</span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">ot including the a=rtcp-mux </span><a href="http://tools.ietf.org/html/draft-ietf-avt-rtp-and-rtcp-mux-07#section-5.1.1" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;" target="_blank">attribute</a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">)</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- Compact SIP headers</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- SDP minimization (removing rtpmap lines, etc)</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- SDP compression</span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">This clearly isn’t an off the shelf SIP and RTP based solution anymore, but how does it all add up? Also, why are three of these efforts focused on minimizing packet size (even if it means violating standards)?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">I know this has been a long and winding road. Hopefully you’re still with me! To understand the value of minimizing packet size you need to peek into another little-known area of the internet: IP fragments. I’m going to butcher a lot of this but at this point you just need to get the broad strokes.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Each network link type is configured for a maximum transmission unit (MTU). For Ethernet this is typically 1500 bytes. This means that the maximum size of a single Ethernet frame can be 1500 bytes. In many, cases, however, the various links and links inside of links (encapsulation), etc mean that the effective end-to-end MTU is significantly smaller than that (we won’t get into ATM, etc). There are two ways this can be addressed:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">- </span><a href="http://en.wikipedia.org/wiki/Path_MTU_Discovery" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">PMTU</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> (Path MTU discovery)</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">- </span><a href="http://en.wikipedia.org/wiki/IP_fragmentation" style="text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">IP fragmentation</span></a><span style="vertical-align: baseline;"></span></span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">With these two technologies who cares about packet size? Firewalls and NAT devices, that’s who. First, many firewall administrators or vendors carelessly block all ICMP packets. That means PMTU is out. If Apple Facetime had greater than end-to-end MTU sized packets and depended on functioning PMTU there would be many instances where it would not work.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">IP fragmentation poses another problem. Once again, many network vendors and firewall administrators outright block IP fragments. To add insult to injury (or is that injury to insult?) many of these devices have broken and/or buggy support for IP fragment reassembly. This especially goes for UDP (which to be fair is not a “connection oriented” protocol). I have personally witnessed many instances where firewall devices could successfully reassemble IP+TCP fragments but not IP+UDP fragments.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">This leads to an almost unwinnable situation. Apple could use TCP for the signalling and UDP for media but then they’d lose the benefits of single port multiplexing. They could use TCP over a single port for everything but that would just be crazy. In either case TCP has a larger header anyway (larger header = larger packet for the same amount of data).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">This has posed such a longstanding problem that some network operators and academics study the behavior of MTU interactions and IP fragmentation on the internet. It came up most recently on </span><a href="http://www.nanog.org/about/home" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">NANOG</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"> a </span><a href="http://seclists.org/nanog/2013/Aug/509" style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; text-decoration: none;" target="_blank"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">few weeks ago</span></a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">. Let’s look at the results of Emile Aben’s mini-study:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Results:</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">size = ICMP packet size, add 20 for IPv4 packet size<br class="kix-line-break" />fail% = % of vantage points where 5 packets where sent, 0 where received.<br class="kix-line-break" />#size fail% vantage points<br class="kix-line-break" />100 0.88 2963<br class="kix-line-break" />300 0.77 3614<br class="kix-line-break" />500 0.88 1133<br class="kix-line-break" />700 1.07 3258<br class="kix-line-break" />900 1.13 3614<br class="kix-line-break" />1000 1.04 770<br class="kix-line-break" />1100 2.04 3525<br class="kix-line-break" />1200 1.91 3303<br class="kix-line-break" />1300 1.76 681<br class="kix-line-break" />1400 2.06 3014<br class="kix-line-break" />1450 2.53 3597<br class="kix-line-break" />1470 3.01 2192<br class="kix-line-break" />1470 3.12 3592<br class="kix-line-break" />1473 4.96 3566<br class="kix-line-break" />1475 4.96 3387<br class="kix-line-break" />1480 6.04 679<br class="kix-line-break" />1480 4.93 3492 [*]<br class="kix-line-break" />1481 9.86 3489<br class="kix-line-break" />1482 9.81 3567<br class="kix-line-break" />1483 9.94 3118</span></div>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">From these results we can see that you start to have more than 1% failure when packet size is greater than </span><strike style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">700</strike><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"> 1000 bytes.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">The Facetime 2010 INVITE packet was 1093 bytes. The Facetime 2013 INVITE packet is 714 bytes. In any case there is a greater chance that the 714 byte packet will reach its intended destination. From these results we can also see that you start to run into real trouble once you reach 1400 bytes. Sure that’s twice the size of our current packet but who knows what the future holds for the capabilities of Facetime? Screen sharing, multi-party conferencing, file transfer, etc all depend on larger SDP descriptions. With the changes Apple has made they’ve not only increased the robustness of Facetime today, they’ve given themselves “room to grow” in the future.</span><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15; vertical-align: baseline;">Apple has spent some time in the trenches over the last few years and found out how difficult real-time communications can be in the real world. </span><span style="font-family: Arial, Helvetica, sans-serif; line-height: 1.15;">Facetime isn't playing around anymore and Apple is becoming a serious networking (services?) company that's posed to take on current best-of-breed solutions.</span></div>
<div dir="ltr" style="margin-bottom: 0pt; margin-top: 0pt;">
<div style="line-height: 1.15;">
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 17px;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; line-height: 17px;">If you're interested to see how this might be implemented in various open source projects, check out the </span><a href="http://blog.krisk.org/2013/09/learning-stupid-nat-tricks-from-apple.html" style="font-family: Arial, Helvetica, sans-serif; line-height: 17px;">update</a><span style="font-family: Arial, Helvetica, sans-serif; line-height: 17px;">!</span></div>
</div>
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span><br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com9tag:blogger.com,1999:blog-9220932811142893057.post-58305753112054052212013-02-20T13:00:00.002-05:002014-03-26T19:13:46.570-04:00findpod - Find Your Own "Packets of Death"<span style="font-family: Arial, Helvetica, sans-serif;"><span id="internal-source-marker_0.03831990408932884" style="vertical-align: baseline;">After my </span><a href="http://blog.krisk.org/2013/02/packets-of-death.html"><span style="color: #1155cc; vertical-align: baseline;">original post</span></a><span style="vertical-align: baseline;"> on “packets of death” I’ve spent the last couple of weeks receiving reports from users all over the world ranging from:</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">- Able to reproduce on 82574L with your “packets of death”</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- Not able to reproduce on 82574L or any other controller</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">-
Not able to reproduce with your “packets of death” but experiencing
identical, sporadic failures across a wide range of controllers</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">Of
course the last category intrigued me the most. There seem to be an
awful lot of people experiencing sporadic failures of their ethernet
controllers but many of them (as I noted in </span><a href="http://blog.krisk.org/2013/02/packets-of-death-update.html"><span style="color: #1155cc; vertical-align: baseline;">another update</span></a><span style="vertical-align: baseline;">)
don’t have the time or tools to diagnose the issue further. In most
cases the symptoms are identical to what I described in my original
post:</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">- Ethernet controller loses link (or reports some other hardware error)</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- Varying amounts of time since boot (hours, days, weeks)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">- Can only be resolved by a reboot or in some cases a complete power cycle</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Many
of these users have been dealing with these failures in various ways
but have been unable to find a root cause. I’ve created a tool to help
them with their diagnosis. It’s called </span><a href="http://www.kriskinc.com/intel-pod" style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #1155cc; vertical-align: baseline;">findpod</span></a><span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"> and it’s been tested on various Debian-based Linux distributions. </span><a href="http://www.kriskinc.com/intel-pod" style="font-family: Arial, Helvetica, sans-serif;"><span style="color: #1155cc; vertical-align: baseline;">Findpod</span></a><span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"> uses the excellent </span><a href="http://0pointer.de/lennart/projects/ifplugd/" style="font-family: Arial, Helvetica, sans-serif;" target="_blank"><span style="color: #1155cc; vertical-align: baseline;">ifplugd daemon</span></a><span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">, the venerable tcpdump, and screen.</span><span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">Once
installed and started iflpugd will patiently wait to receive link
status notifications from the Linux kernel. Once link is detected on
the target interface it will start a tcpdump session running inside of
screen. This tcpdump session will log all packets sent and received on
that interface. Here’s the thing - many of these failures are reported
after days or weeks of processed traffic - the tcpdump capture file
could easily reach several gigabytes or more! Here’s where one key
trick in findpod comes into play - by default findpod will only log the
last 100MB sent or received on the target interface. As long as ifplugd
doesn’t report any link failures tcpdump will keep writing to the same
100MB circular capture file.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">What
happens when the interface loses link? This is the second unique feature of findpod. When ifplugd reports a loss of link it will wait for 30
seconds before stopping the packet capture and moving the capture file
to a meaningful (and known) name. If you think your ethernet controller
failures could be related to the types of traffic you’re sending or
receiving (as I discovered with my “packets of death”) findpod will help
you narrow it down to (at most) 100MB of network traffic, even if the
capture runs for weeks and your interface handles GBs of data!</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">Of
course (more than likely) it’s even easier than that; if your link loss
is being caused by a specific received packet it will be the last
packet in the capture file provided by findpod and you’ll only have a
100MB capture file to work with. If your issue is anything like mine you should be able to isolate it down to a specific packet that you can feed to <a href="http://tcpreplay.synfin.net/" target="_blank">tcpreplay</a>; reproducing your controller issue on demand.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">Please tell me about your experiences with </span><a href="http://www.kriskinc.com/intel-pod"><span style="color: #1155cc; vertical-align: baseline;">findpod</span></a><span style="vertical-align: baseline;">. As always, comments and suggestions are welcome!</span></span>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-20899758468979685932013-02-08T15:42:00.003-05:002014-03-26T19:14:13.126-04:00Packets of Death - UPDATE<span style="font-family: Arial, Helvetica, sans-serif; font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">UPDATE - Intel has pointed me towards <a href="http://communities.intel.com/community/wired/blog/2012/10/18/i210-launch-announcement" target="_blank">the successor to the 82574</a>, which includes some of the features I suggest here. I think it's safe to say we're all looking forward to this chip hitting the streets!</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span id="internal-source-marker_0.3670935509726405"><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">The last 48 hours has been interesting, to say the least.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">My </span><a href="http://blog.krisk.org/2013/02/packets-of-death.html"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">original post</span></a><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"> gathered much more attention than I originally thought. I’ll always remember being on a conference call and having someone tell me “Hey, you’re on </span><a href="http://tech.slashdot.org/story/13/02/06/2024251/intel-gigabit-nic-packet-of-death"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">Slashdot</span></a><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">”. Considering the subject matter I suppose I should have expected that.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">As of today, here’s what I know:</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">Many of you have shared the results of your testing. The vast majority of tested Intel ethernet controllers do not appear to be affected by this issue.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">Intel has </span><a href="http://communities.intel.com/community/wired/blog/2013/02/07/intel-82574l-gigabit-ethernet-controller-statement"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">responded</span></a><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"> with an expanded technical explanation of the issue. I also received a very pleasant and professional phone call from Douglas Boom (at Intel) to update me on their assessment of the situation and discuss any ongoing concerns or problems I may have. Thank you Doug and well done Intel! Note to other massive corporations that could be presented with issues like this: do what Intel did.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">To summarize their response, Intel says that a misconfigured EEPROM caused this issue. The EEPROM is written by the motherboard manufacturer and not Intel. Intel says my motherboard manufacturer did not follow their published guidelines for this process. Based on what I’ve seen, how my issue was fixed, and what I’m learning from the crowdsourced testing process this seems like a perfectly plausible explanation. Once again, thanks Intel!</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">However, I still don’t believe this issue is completely isolated to this specific instance and one motherboard manufacturer. For one, I have received at least two confirmed reports from people who were able to reproduce this issue - my “packet of death” shutting down 82574L hardware from different motherboard manufacturers. This doesn’t surprise me at all.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">One thing we’re reminded of in this situation is just how complex all of these digital systems have become. We’re a long way from configuring ethernet adapters with IRQ jumpers. Intel has designed an incredibly complex ethernet controller - the </span><a href="http://www.intel.com/content/www/us/en/ethernet-controllers/82574l-gbe-controller-datasheet.html"><span style="color: #1155cc; vertical-align: baseline; white-space: pre-wrap;">datasheet</span></a><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"> is 490 pages long! Of course I’m not faulting them for this - the features available in this controller (or many other controllers) dictate this level of complexity - management/IPMI features, WOL, various offloading mechanisms, interrupt queues, and more. This complexity doesn’t even scratch the surface of the various other systems involved in getting data across the internet and into your eyeballs!</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">Like any sufficiently advanced product all of these features are driven by a configuration mechanism. The Linux kernel module for the 82574L (e1000e) has various options that can be passed to modify the behavior of the adapter. Makes sense. If I passed some stupid or unknown parameter to this module I would expect it to return with some kind of error informing me of my mistake. I’m only human, mistakes are going to happen.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">At a lower level Intel has exposed these EEPROM configuration parameters to control various aspects of the controller. As Intel says these EEPROM values are to be set by the motherboard manufacturer. Here’s where the problem lies - it’s certainly possible this could be done incorrectly. Motherboard manufacturers are human and they make mistakes too.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">Unfortunately, as we’ve learned in this case, there isn’t quite the same level of feedback when EEPROM misconfigurations happen. In my previous example if I pass unknown parameters to a kernel module it’s going to come back and say “Hey - I don’t know what that is (dummy)” and exit with an error.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">As I’ve shown in some cases (mine) if an EEPROM is misconfigured everything appears normal until some insanely specific packet is received. Then the controller shuts down, for some reason.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">Does that behavior make sense to anyone?</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">I suggest the following:</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">1) Make future controllers have as much in-hardware sane behavior as possible when unknown conditions are encountered. Error checking, basically. Users can input data on a web form, that’s why there’s error checking. Everyone knows users do stupid things. Clearly some of the people programming Intel EEPROMs for motherboard OEMs do stupid things too. What is sane default behavior? EEPROM error encountered = adapter shutdown and error message. Give the user notification and provide some mechanism for EEPROM validation and management...</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">2) Put more EEPROM validation in operating system drivers. Intel maintains ethernet drivers for various platforms. Why aren’t these drivers doing more validation of the adapter EEPROM? If my EEPROM was so badly misconfigured, why couldn’t the e1000e module have discovered that and notified me?</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">3) Produce and support an external tool for EEPROM testing, programming, and updating. In the course of working with Intel last fall I was provided a version of this tool for my testing so I know it exists. While I can understand why you don’t want random users messing with their EEPROM (and causing potential support nightmares) it seems the benefits would clearly outweigh any potential problems (of which there are already plenty).</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">The reality is Intel has no idea how many systems are affected by this issue or could be affected by issues like it. How could they? They’re expecting motherboard OEMs to follow their instructions (and understandably so). Just look at the combination of variables required to reproduce this issue:</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">- Intel 82574L</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">- Various specific misconfigured bytes in the EEPROM</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">- An insanely specific packet with the right value at just the right byte, received at a specific time</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">While most people weren’t able to reproduce this issue with their controller and EEPROM combinations it did kick off various discussions of periodic, random, sporadic failures across a wide range of ethernet adapters and general computing weirdness. A quick Google search returns a wide assortment of complaints with these adapters (and others like it) from a whole slew of users. EEPROM corruption. Random adapter resets. Packet drops. Various latency issues. PCI bus scan errors. ASPM problems. The list goes on and on.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">Perhaps the “packet of death” for a slightly misconfigured Intel 82579 (for example) is my packet shifted 20 bytes in one direction or the other. Who knows? Please, please, please Intel - lets do everyone a favor and get these EEPROMs under control. End users update firmware all of the time - routers, set-top boxes, sometimes even their cars! Why can’t we have some utility to make sure our ethernet adapters aren’t just waiting to freak out when they receive the wrong packet?</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">I don’t believe in magic, swamp gas, sun spots, or any of the other “explanations” offered for some of the random strange behavior we often see with these complex devices (ethernet adapters or otherwise). That’s why I spent so long working on this issue to find a root cause (well that and screaming customers). I, like anyone else, encounter bugs and general weirdness in devices and software everyday in my life. Most of the time how do I respond to these bugs? I reboot, shrug my shoulders, say “that was weird”, and move on. Meanwhile I know, deep down, that there is a valid explanation for what just happened. Just like there was with my ethernet controllers.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">Even with the explanation offered by Intel we could go much deeper. Why these bytes at that location? Why this packet? What’s up with the “inoculation” effect of some of the values? There are still many unanswered questions.</span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">I’ve enjoyed reading many others report their tales of “extreme debugging” with the digital devices in their lives. It seems I’m not the only one that isn’t always satisfied with saying “that was weird” and moving on.</span></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-weight: normal; text-decoration: initial; vertical-align: baseline; white-space: pre-wrap;">I've said it before and I'll say it again - I love the internet!</span></span>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com4tag:blogger.com,1999:blog-9220932811142893057.post-77833149944592106372013-02-06T11:01:00.000-05:002014-03-26T19:14:39.790-04:00Packets of Death<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">When you're done reading this <a href="http://blog.krisk.org/2013/02/packets-of-death-update.html">check out my update</a>! Experiencing similar ethernet controller issues but <a href="http://blog.krisk.org/2013/02/findpod-find-your-own-packets-of-death.html">don't know where to start</a>?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span>
<span style="vertical-align: baseline;">UPDATE: See the packets and find out if you're affected <a href="http://www.kriskinc.com/intel-pod" target="_blank">here</a>.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span>
<span style="vertical-align: baseline;">UPDATE 2: Yes, I've reproduced this issue regardless of OS, ASPM state/settings, or software firewall settings. Obviously if you have a layer 2/3 firewall in front of an affected interface you'll be ok.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"><br /></span>
<span id="internal-source-marker_0.01523004649198012" style="vertical-align: baseline;">Packets of death. I started calling them that because that’s exactly what they are.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Star2Star
has a hardware OEM that has built the last two versions of our
on-premise customer appliance. I’ll get more into this appliance and the
magic it provides in another post. For now let’s focus on these killer
packets.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">About
a year ago we released a refresh of this on-premise equipment. It
started off simple enough, pretty much just standard Moore’s Law stuff.
Bigger, better, faster, cheaper. The new hardware was 64-bit capable,
had 8X as much RAM, could accommodate additional local storage, and had
four Intel (my preferred ethernet controller vendor) gigabit ethernet
ports. We had (and have) all kinds of ideas for these four ports. All in
all it was pretty exciting.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">This
new hardware flew through performance and functionality testing. The
speed was there and the reliability was there. Perfect. After this
extensive testing we slowly rolled the hardware out to a few beta sites.
Sure enough, problems started to appear.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">All it takes is a quick </span><a href="https://www.google.com/#hl=en&output=search&sclient=psy-ab&q=82574l+e1000e&oq=82574l+e1000e&gs_l=hp.3...1193.4410.0.4568.13.13.0.0.0.0.80.872.13.13.0.les%3B..0.0...1c.1.jku0hV5K7oU&pbx=1&bav=on.2,or.r_gc.r_pw.r_cp.r_qf.&fp=4abeb2e72661de26&biw=1855&bih=994"><span style="color: #1155cc; vertical-align: baseline;">Google search</span></a><span style="vertical-align: baseline;">
to see that the Intel 82574L ethernet controller has had at least a few
problems. Including, but not necessarily limited to, EEPROM issues,
ASPM bugs, MSI-X quirks, etc. We spent several months dealing with each
and every one of these. We thought we were done.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">We weren’t. It was only going to get worse.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">I
thought I had the perfect software image (and BIOS) developed and
deployed. However, that’s not what the field was telling us. Units kept
failing. Sometimes a reboot would bring the unit back, usually it
wouldn’t. When the unit was shipped back, however, it would work when
tested.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Wow. Things just got weird.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">The
weirdness continued and I finally got to the point where I had to roll
my sleeves up. I was lucky enough to find a very patient and helpful
reseller in the field to stay on the phone with me for three hours while
I collected data. This customer location, for some reason or another,
could predictably bring down the ethernet controller with voice traffic
on their network.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Let
me elaborate on that for a second. When I say “bring down” an ethernet
controller I mean BRING DOWN an ethernet controller. The system and
ethernet interfaces would appear fine and then after a random amount of
traffic the interface would report a hardware error (lost communication
with PHY) and lose link. Literally the link lights on the switch and
interface would go out. It was dead.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Nothing
but a power cycle would bring it back. Attempting to reload the kernel
module or reboot the machine would result in a PCI scan error. The
interface was dead until the machine was physically powered down and
powered back on. In many cases, for our customers, this meant a truck
roll.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">While
debugging with this very patient reseller I started stopping the packet
captures as soon as the interface dropped. Eventually I caught on to a
pattern: the last packet out of the interface was always a 100 Trying
provisional response, and it was always a specific length. Not only
that, I ended up tracing this (Asterisk) response to a specific phone
manufacturer’s INVITE.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">I
got off the phone with the reseller, grabbed some guys and presented my
evidence. Even though it was late in the afternoon on a Friday,
everyone did their part to scramble and put together a test
configuration with our new hardware and phones from this manufacturer.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">We
sat there, in a conference room, and dialed as fast as our fingers
could. Eventually we found that we could duplicate the issue! Not on
every call, and not on every device, but every once in a while we could
crash the ethernet controller. However, every once in a while we
couldn’t at all. After a power cycle we’d try again and hit it. Either
way, as anyone who’s tried to diagnose a technical issue knows the first
step is duplicating the problem. We were finally there.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Believe
me, it took a long time to get here. I know how the OSI stack works. I
know how software is segmented. I know that the contents of a SIP packet
shouldn’t do anything to an ethernet adapter. It just doesn’t make any
sense.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Between
packet captures on our device and packet captures from the mirror port
on the switch we were finally able to isolate the problem packet. Turns
out it was the received INVITE, not the transmitted 100 Trying! The
mirror port capture never saw the 100 Trying hit the wire.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Now
we needed to look at this INVITE. Maybe the userspace daemon processing
the INVITE was the problem? Maybe it was the transmitted 100 Trying?
One of my colleagues suggested we shutdown the SIP software and see if
the issue persisted. No SIP software running, no transmitted 100 Trying.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">First we needed a better way to transmit the problem packet. We isolated the INVITE transmitted from the phone and used </span><a href="http://tcpreplay.synfin.net/"><span style="color: #1155cc; vertical-align: baseline;">tcpreplay</span></a><span style="vertical-align: baseline;">
to play it back on command. Sure enough it worked. Now, for the first
time in months, we could shut down these ports on command with a single
packet. This was significant progress and it was time to go home, which
really meant it was time to set this up in the lab at home!</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;">Before I go any further I need to give another shout out to an excellent open source piece of software I found. </span><a href="http://code.google.com/p/ostinato/"><span style="color: #1155cc; vertical-align: baseline;">Ostinato</span></a><span style="vertical-align: baseline;">
turns you into a packet ninja. There’s literally no limit to what you
can do with it. Without Ostinato I could have never gotten beyond this
point.</span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">With my packet Swiss army knife in hand I started poking and prodding. What I found was shocking.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">It all starts with a strange SIP/SDP quirk. Take a look at this SDP:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">v=0</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">o=- 20047 20047 IN IP4 10.41.22.248</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">s=SDP data</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">c=IN IP4 10.41.22.248</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">t=0 0</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">m=audio 11786 RTP/AVP 18 0 18 9 9 101</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=rtpmap:18 G729/8000</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=fmtp:18 annexb=no</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=rtpmap:0 PCMU/8000</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=rtpmap:18 G729/8000</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=fmtp:18 annexb=no</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=rtpmap:9 G722/8000</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=rtpmap:9 G722/8000</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=fmtp:101 0-15</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=rtpmap:101 telephone-event/8000</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=ptime:20</span><br />
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=sendrecv</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; color: #222222; vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="background-color: white; color: #222222; vertical-align: baseline;">Wireshark picture:</span><img height="328px;" src="https://lh3.googleusercontent.com/8ROejWpeKrGEVRjNbR1oZB_Ru66Ng2iPbMK-Lm1xcKhWOtoTOpH7mS3C3kiT3AS20arO_6GH-9VShBiXMuxct6axOLoaed6Nw4gSdbBkGG7CCYCciVCk" width="657px;" /><span style="background-color: white; color: #222222; vertical-align: baseline;"></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Yes,
I saw it right away too. The audio offer is duplicated and that’s a
problem but again, what difference should that make to an Ethernet
controller?!? Well, if nothing else it makes the ethernet frame
larger...</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">But
wait, there were plenty of successful ethernet frames in these packet
captures. Some of them were smaller, some were larger. No problems with
them. It was time to dig into the problem packet. After some more
Ostinato-fu and plenty of power cycles I was able to isolate the problem
pattern (with a problem frame).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Warning: we’re about to get into some hex.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">The
interface shutdown is triggered by a specific byte value at a specific
offset. In this case the specific value was hex 32 at 0x47f. Hex 32 is
an ASCII 2. Guess where the 2 was coming from?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="background-color: white; color: #222222; font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">a=ptime:20</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">All
of our SDPs were identical (including ptime, obviously). All of the
source and destination URIs were identical. The only difference was the
Call-IDs, tags, and branches. Problem packets had just the right
Call-ID, tags, and branches to cause the “2” in the ptime to line up
with 0x47f.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">BOOM!
With the right Call-IDs, tags, and branches (or any random garbage) a
“good packet” could turn into a “killer packet” as long as that ptime
line ended up at the right address. Things just got weirder.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">While
generating packets I experimented with various hex values. As if this
problem couldn’t get any weirder, it does. I found out that the behavior
of the controller depended completely on the value of this specific
address in the first received packet to match that address. It broke
down to something like this:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"> Byte 0x47f = 31 HEX (1 ASCII) - No effect</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"> Byte 0x47f = 32 HEX (2 ASCII) - Interface shutdown</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"> Byte 0x47f = 33 HEX (3 ASCII) - Interface shutdown</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;"> Byte 0x47f = 34 HEX (4 ASCII) - Interface inoculation</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Bad:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><img height="137px;" src="https://lh5.googleusercontent.com/gF1gg-IzbCVuy_PfJcFw04nexSp9iD5_KFIbkEHZl_nXBnPZZHkTHYuJuGTZ0y3w9iTBs0uKCyNwPoNBzPJvTSm5spQeMKq6NgAf6-c6PHOdrumyk_L5" width="600px;" /><span style="vertical-align: baseline;"></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">Good:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><img height="138px;" src="https://lh5.googleusercontent.com/GMmEtySLYgYklecvJ7jU9Cp1ivZPUt_UpoSbqG6zJfDczQ8VWQ1QxMnpPRNS1GwBdSKNgZajLjdRHzjYjlQ4xVtGF85cAJnWdENtpXWpZMitM2YdMaRz" width="596px;" /><span style="vertical-align: baseline;"></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">When
I say “no effect” I mean it didn’t kill the interface but it didn’t
inoculate the interface either (more on that later). When I say the
interface shutdown, well, remember my description of this issue - the
interface went down. Hard.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">With
even more testing I discovered this issue with every version of Linux I
could find, FreeBSD, and even when the machine was powered up
complaining about missing boot media! It’s in the hardware; the OS has
nothing to do with it. Wow.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">To
make matters worse, using Ostinato I was able to craft various versions
of this packet - an HTTP POST, ICMP echo-request, etc. Pretty much
whatever I wanted. With a modified HTTP server configured to generate
the data at byte value (based on headers, host, etc) you could easily
configure an HTTP 200 response to contain the packet of death - and kill
client machines behind firewalls!</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">I
know I’ve been pointing out how weird this whole issue is. The
inoculation part is by far the strangest. It turns out that if the first
packet received contains any value (that I can find) other than 1, 2,
or 3 the interface becomes immune from any death packets (where the
value is 2 or 3). Also, valid ptime attributes are defined in <strike>powers</strike> (edit: multiples) of
10 - 10, 20, 30, 40. Depending on Call-ID, tag, branch, IP, URI, etc
(with this buggy SDP) these valid ptime attributes line up perfectly.
Really, what are the chances?!?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">All
of a sudden it’s become clear why this issue was so sporadic. I’m
amazed I tracked it down at all. I’ve been working with networks for
over 15 years and I’ve never seen anything like this. I doubt I’ll ever
see anything like it again. At least I hope I don’t...</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">I
was able to get in touch with two engineers at Intel and send them a
demo unit to reproduce the issue. After working with them for a couple
of weeks they determined there was an issue with the EEPROM on our
82574L controllers.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">They
were able to provide new EEPROM and a tool to write it out.
Unfortunately we weren’t able to distribute this tool and it required
unloading and reloading the e1000e kernel module, so it wouldn’t be
preferred in our environment. Fortunately (with a little knowledge of
the EEPROM layout) I was able to work up some bash scripting and ethtool
magic to save the “fixed” eeprom values and write them out on affected
systems. We now have a way to detect and fix these problematic units in
the field. We’ve communicated with our vendor to make sure this fix is
applied to units before they are shipped to us. What isn’t clear,
however, is just how many other affected Intel ethernet controllers are
out there.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="vertical-align: baseline;"></span><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; vertical-align: baseline;">I guess we’ll just have to see...</span>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com105tag:blogger.com,1999:blog-9220932811142893057.post-22157022952223941862012-09-21T20:18:00.003-04:002012-09-21T20:18:53.212-04:00FreeSWITCH Stable Branch - Sponsored by Star2Star<b id="internal-source-marker_0.3357964907772839" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As many of you know I’ve been quite fond of </span><a href="http://www.freeswitch.org/"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">FreeSWITCH</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for some time now. I’ve been impressed with the functionality, stability, and performance. Did I say impressed? I suppose I meant to say thrilled. Some of my more long-term readers may remember </span><a href="http://blog.krisk.org/2009/10/i-dont-do-testimonials.html"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">this post</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from three years ago. I can’t believe it’s been that long. I suppose that makes me (officially) old.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Getting back on track (as I often have to do) I have had one concern with FreeSWITCH over the years - the lack of a stable branch. Traditionally (in software development) once a project or piece of code reaches a certain level of maturity (or use, even) there comes a need to segment the code into an independently maintained entity. In most revision control systems this is typically called a “branch”. Makes sense because lines of code are often referred to as “trees”. I get it!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Since the beginning of the project in 2006 all of FreeSWITCH development has take place in one central code repository (trunk - trees again). At any given point in time a user could pull down trunk and receive the latest and “greatest” code - new features, bug fixes, security patches, etc. Unfortunately these new features can introduce new bugs. They can re-expose old bugs. They can change behavior in unexpected ways.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One of the things that has been most impressive about the FreeSWITCH project, actually, has been the relative stability of this often new and untested code. It’s been extremely rare to find a serious issue in trunk. However, the mere knowledge of this code management practice (or relative lack thereof) and potential for new features/bugs is an issue.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I learned a long time ago that people expect their phones to just work. The same user who finds it perfectly acceptable to reboot their computer after a software crash will jump and scream when a telephone doesn’t work. I guess that’s a testament to the level of experience people have historically expected. Interestingly this expectation seems to be changing in the face of new technology, which is </span><a href="http://blog.krisk.org/2012/02/hyperspecialization-and-shakeup-of-100.html"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">something I have covered before</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Anyway, for the time being people (including myself) expect these things to “just work”. Of course this expectation places a tremendous burden on network and facility operators such as myself (and rightfully so). Meeting this expectation with FreeSWITCH trunk has required a significant amount of testing, testing, re-testing, regression testing, patching, etc. We simply cannot unleash a new version of software in the field without significant testing. When issues are found we have to address them. This takes time and resources.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Meanwhile FreeSWITCH is an open source project. That can’t be expected to provide for every need and whim of large commercial users such as myself. Why should they? They already spend so much of their limited time and resources to provide software that essentially powers my business and (indirectly) provides for my livelihood - for free. With these limited resources alone they can’t be expected to make time for the maintenance of yet another collection of code.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">That’s why I’m happy to announce the Star2Star sponsorship of the FreeSWITCH stable branch! For the past several months Star2Star has been providing the financial assistance necessary for the FreeSWITCH project to hire another full-time team member to not only maintain a stable branch (1.2 as of this writing) but improve documentation, packaging, and community interaction. Everyone at Star2Star (including myself) couldn’t be happier to provide this resource for the project and the community. We look forward to working more with the FreeSWITCH team on the stable branch and any other projects that may advance FreeSWITCH and the state of the art in communications!</span></b>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com2tag:blogger.com,1999:blog-9220932811142893057.post-61565924475775499072012-07-26T13:48:00.000-04:002012-07-26T13:48:36.885-04:00Performance Testing (Part 2) - Going CommercialSome time ago I wrote the <a href="http://blog.krisk.org/2011/12/performance-testing-part-1.html">first installment</a> of a multi-part series on performance testing. Here we go with part two!<br />
<br />
<b id="internal-source-marker_0.8320313754957169" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In part one I talked about some of the difficulties surrounding performance testing - functionality, flexibility, high scale, quality metrics, etc.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After looking at a couple commercial products we discovered a few things:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1) Some of these products really do “have it all”.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2) They can be very expensive.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There was some initial sticker shock ($100,000+) but looking back I shouldn’t have been so surprised. My first reaction (of course) was to reach out to a few people in the open source community with a proposal. In a classic “build vs. buy” scenario I wanted to build. This is (roughly) what I needed:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- At least 40,000 RTP streams (20,000 calls)</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- At least 100,000 SIP calls and/or registrations</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- The ability to emulate multiple user agents (VLAN, MAC address, DHCP client, SIP UA)</span></b><br />
<b id="internal-source-marker_0.8320313754957169" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- RTP stream analysis on each RTP leg (MOS, R-Factor, PESQ, etc)</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Flexible device emulation - SIP headers, supported SIP features, etc</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Multiple codec support (at least G711, G729, G722, SILK/OPUS with FEC et</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">all, etc).</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Control of test scenarios - CPS, number of calls, duration of call,</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">total duration of test, etc</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Ability to save/load tests via web interface (for ease of use,</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">comparison of results, etc)</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Ability to perform feature testing - generate DTMF during a call to</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">navigate an IVR, for example.</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- A modular system for monitoring the device under test - Linux load,</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">CPU usage, disk usage, network I/O, etc. Could also monitor Cisco</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">switches in between devices, Windows hosts, etc. Maybe even</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">FreeSWITCH or Asterisk if that's what was running on the device under</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">test.</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Saving and graphing of all relevant performance data - call setup</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">time, delay, duration, RTP jitter, packet loss, RTP stream stats, etc.</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Ability to save data and generate reports from said data.</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Scalable design with master/slave architecture to scale across hosts</span><br /><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">with the ability for hardware.</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Did I mention this tool needs to be usable by various test engineers, some of which don’t know the difference between SIP and SDP (and rightfully so, they shouldn’t need to)?</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">With the open source software already available I figured this could be made available for less than the cost of a commercial testing solution.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I gave it away with the title of this post but you can guess what happened next: it was going to cost far more to develop everything I needed. By the way - it would also take six months to build and take 10 years off my life hunting bugs, managing the project, etc.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">BUY</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For less than what it would cost to build everything above I could buy </span><a href="http://www.ixiacom.com/products/xcellon/xcellon_ultra_np_load_module/index.php"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">this</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. A multi-user chassis with one Xcellon-Ultra NP load module and room for another one. 180,000 emulated SIP endpoints. 96,000 RTP streams. Wire speed 10 gigabit VoIP testing (and 12 gigabit ports).</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Of course this isn’t a perfect world... The chassis runs Windows. The client software is only available for Windows and the interface is probably the furthest from what I want. As a guy that eats, lives, and breathes CLI (and has for a decade) multi-pane/dropdown/hidden/shadow GUIs are NOT my thing. I don’t even know what to call or how to describe some of the window/GUI elements present in the IxLoad user interface...</span></b><br />
<br />
<b id="internal-source-marker_0.8320313754957169" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ixia even has a solution to this problem. They offer TCL scripting with clients for various platforms, including Linux! While we’ll eventually get into that for the time being we went with a much simpler solution: we setup a Windows terminal server. I use </span><a href="http://cord.sourceforge.net/"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">CoRD</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> from my Mac, login to the terminal server, and run a test. As you’ll see in part three - IT JUST WORKS.</span></b><b id="internal-source-marker_0.8320313754957169" style="color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></b><b id="internal-source-marker_0.8320313754957169" style="color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: white; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></b>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-17658283303480623192012-07-12T13:10:00.001-04:002012-07-12T13:10:39.243-04:00My Linux Story<b id="internal-source-marker_0.12979812803678215" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">While digging through some boxes the other day I came across a book. Not just any book - a very important book. A book I both cursed and loved more than any other book in my entire life.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To tell the story of the book I need to give a little background. My father was a professor at the University of Illinois at Chicago. When I was old enough (13) I would spend my summers volunteering in his department (Occupational Therapy) doing random IT jobs - making ethernet cables, cleaning up viruses, fixing printers, etc.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One day they took me on a tour of the other departments in his </span><a href="http://www.ahs.uic.edu/ahs/about/history.php" target="_blank"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">building</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. They had some medical visualization apps running in one of the departments. Here we were, sitting in a room, with some PhD students working on some of the coolest computers I’d ever seen. They had bright colors and huge monitors. They ran tons of advanced looking applications. Most of the interaction was through a very interesting command line interface. I was intrigued.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Being a rambunctious 13 year old I approached one of the students and complimented him on his computer. I also asked him how I could get one of them for my house. He laughed. At the time I didn’t understand why it was a funny question. Looking back now I know it’s because those “computers” were actually high powered Silicon Graphics Indigo workstations that cost upwards of $40,000 each (in 1997 dollars).</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">His response was a muffled “Ummm, you can’t”. Not being dissuaded my follow-up question was “How can I get something like it?”. One of the students looked at the other student. They both kind of shrugged until one of them said “umm, Linux?”. Still not being quite sure they both agreed that “Linux” would be my best bet. I asked them how I could get “Linux”. They told me to go down to the university book store and buy a book about Linux. A good Linux book, they said, should include a CD-ROM with a copy of Linux I could install on my computer at home.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">That day, before we went to Union Station to catch the Metra, my dad and I went to the university bookstore to find a book about Linux. Sure enough we found a Linux book that included not one but TWO CD-ROMs! I read as much of the book as I could before I got home that night.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once I was at home I booted the family computer (with a DOS bootdisk) and ran a loadlin (wow, remember that?) based install program from the CD-ROM (no el torito!). During the course of the install I repartitioned and reformatted the (only) family computer - an IBM Aptiva M51. It had a Pentium 100, 1GB hard drive, and 16MB of RAM (upgraded from 8MB). It also came with one of those HORRIBLE MWAVE adapters and some shady Cirrus Logic graphics (IIRC).</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Anyway, the install process (which was pretty horrible, actually) left me with what was (at the time) a barely usable computer. How was my sister going to do her homework? How were we going to get on to the internet? Uh-oh, looks like I better learn more about this “Linux” thing...</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So that’s how it started. Later that year (for Christmas) my parents realized they weren’t going to get the Aptiva back from me so we bought another computer to run Windows. At that point my little Linux workstation became my computer and the “gateway” for my first home network - a 10BASE2 (coax!) Ethernet network using NE2000 cards from Linksys. Internet access went through my Aptiva using an external modem that, regardless of type, could only negotiate a maximum of 19,200 bps on our crappy phone line. PPP, chat scripts, dial-on-demand, squid for caching, etc. 15 years later I’m still (primarily) using Linux!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After finding “My First Linux Book” I wondered what would happen if I tried to install that version of Linux today. Some people reminisce about their childhood by hearing certain songs, playing a sport, or collecting action figures. I (apparently) do it by installing ancient versions of Linux.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I needed to install this version of Linux but the CD-ROMs in my book were missing. I looked around the internet for a while but could not find an ISO or copy of the distro anywhere. I could barely find references to the book. Where else could I look? Everywhere else I look - Amazon!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sure enough, Amazon had a </span><a href="http://www.amazon.com/Linux-Universe-Configuration-Stefan-Strobel/dp/0387948791/ref=sr_1_1?ie=UTF8&qid=1342103282&sr=8-1&keywords=Linux+Universe" target="_blank"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">used copy of the book</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (with CD-ROM) for $5 with shipping. Two days later it was here. To my surprise the book (and the CDs) were in excellent condition. Who the hell is keeping a warehouse full of mid-nineties Linux books (3rd edition with Kernel 2.0!)?</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I get on my main development machine at home, download a FreeDOS ISO (to install from, remember), and create a VirtualBox virtual machine. What should it look like? I decide to “go big”:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><b id="internal-source-marker_0.12979812803678215" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1 CPU</span></b></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><b id="internal-source-marker_0.12979812803678215" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">256MB of RAM</span></b></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><b id="internal-source-marker_0.12979812803678215" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8GB Intel PIIX3 hard drive</span></b></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><b id="internal-source-marker_0.12979812803678215" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1 Am79C973 ethernet port</span></b></li>
</ul>
<b id="internal-source-marker_0.12979812803678215" style="-webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; color: black; font-family: Times; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Keep in mind I’m going to be running kernel 2.0 here - this hardware needs to be supported by a 15 year old kernel. I get </span><a href="http://www.unifix.de/products/unifix_2_0/" target="_blank"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Unifix Linux 2.0</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> installed. Moments later I’m logged into my “new” Linux system. Not knowing exactly what to do now, I decide to try to get networking to work.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Long story short I could not get Linux 2.0 to recognize the emulated Am79C973 ethernet controller. I tried changing the device ids and recompiling the kernel (takes less than one minute, btw) but couldn’t get it to work.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Hmmm, what else could I do for connectivity? Maybe I could go really nostalgic and get something running over the serial port?</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I configured VirtualBox to emulate a 16550 serial port as COM1. I setup VirtualBox to point the other end of the emulated serial port to a local pipe. I figured that if I could somehow run pppd on both sides of this serial port (host and guest) and configure NAT I could get this thing on the internet.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here’s how I did it:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1) Launch socat to convert the unix domain socket provided by VirtualBox to a standard Linux tty so pppd can run on it:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(on host)</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">socat UNIX-CONNECT:[com1] PTY,link=[vmodem0],raw,echo=0,waitslave</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Where [com1] is the path to your VirtualBox socket and [vmodem0] is the path to your (new) tty.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2) Launch pppd on the new tty:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(on host)</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pppd [vmodem0] 57600 192.168.100.1:192.168.100.2 nodetach local</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once again where [vmodem0] is the path to your new socat tty. Make sure that the IP addresses provided for each end of the PPP link don’t conflict with any local IP addresses.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3) Setup kernel iptables on the host:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(on host)</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">echo 1 > /proc/sys/net/ipv4/ip_forward</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">iptables -A FORWARD -i eth0 -o ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">iptables -A FORWARD -i ppp0 -o eth0 -j ACCEPT</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" />4) Connect the virtual machine to the host:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(on guest)</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">pppd /dev/ttyS0 57600 defaultroute passive</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Sure enough, here’s what I saw on the host:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Using interface ppp0</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Connect: ppp0 <--> /home/kris/projects/linux_universe/vmodem0</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">local IP address 192.168.100.1</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">remote IP address 192.168.100.2</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Boom! 1990s Linux, meet the 21st century!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once I had networking up and running things really took off. I noticed all of the services running by default on my old Linux host (portmap, yp, apache, telnet, echo, chargen, sendmail, wu-ftpd, etc). Remember the 90s when the internet wasn’t such a hostile place!?!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here’s some fun command output from my “new” host old_linux:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:~ # uname -a</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Linux old_linux 2.0.25 Unifix-2.0 i686</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:~ # ping -c 5 192.168.100.1</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PING 192.168.100.1 (192.168.100.1): 56 data bytes</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=15.1 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=19.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=19.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=19.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=19.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">--- 192.168.100.1 ping statistics ---</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5 packets transmitted, 5 packets received, 0% packet loss</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">round-trip min/avg/max = 15.1/18.9/19.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:~ #</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">20ms of latency on the same physical machine (pppd -> VirtualBox -> socat -> pppd)!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:~ # ping -c 5 www.google.com</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">PING www.l.google.com (74.125.139.104): 56 data bytes</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 74.125.139.104: icmp_seq=0 ttl=45 time=59.5 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 74.125.139.104: icmp_seq=1 ttl=45 time=49.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 74.125.139.104: icmp_seq=2 ttl=45 time=49.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 74.125.139.104: icmp_seq=3 ttl=45 time=50.5 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">64 bytes from 74.125.139.104: icmp_seq=4 ttl=45 time=69.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">--- www.l.google.com ping statistics ---</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5 packets transmitted, 5 packets received, 0% packet loss</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">round-trip min/avg/max = 49.9/55.9/69.9 ms</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:~ # gcc -v</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Reading specs from /usr/lib/gcc-lib/i486-unknown-linux/2.7.2.1/specs</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">gcc version 2.7.2.1</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:~ # httpd -v</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Server version Apache/1.1.1.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:~ # ssh -v</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SSH Version 1.2.14 [i486-unknown-linux], protocol version 1.4.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Standard version. Does not use RSAREF.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Pre-iptables. Pre-ipchains. IPFWADM!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:~ # ipfwadm -h</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ipfwadm 2.3.0, 1996/07/30</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:/usr/src/linux-2.0.25 # time make</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">[trimmed output]</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The new kernel is in file arch/i386/boot/bzImage; now install it as vmlinuz</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A name list of the kernel is in vmlinux.map; you may install it as vmlinux</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">9.86user 11.74system 0:22.78elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0inputs+0outputs (0major+0minor)pagefaults 0swaps</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:/usr/src/linux-2.0.25 #</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">23 seconds to compile the kernel!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Ok, but what about the modules?</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">make[1]: Leaving directory `/usr/src/linux-2.0.25/arch/i386/math-emu'</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">8.80user 7.42system 0:16.80elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">0inputs+0outputs (0major+0minor)pagefaults 0swaps</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">root@old_linux:/usr/src/linux-2.0.25 #</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16 seconds to compile the modules!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have to admit I’m fascinated with this distro and the state of Linux as I was first introduced to it. Of course some of the memories have faded over time but more than anything it’s amazing to see how far we’ve come...</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1997:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Linux 2.0.25</span></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Original Pentium at 100MHz</span></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">16MB RAM</span></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1GB hard drive</span></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1MB VRAM</span></li>
</ul>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2012:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Linux 3.2.0 (Ubuntu 12.04)</span></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Quad-core Pentium Xeon at 2.27GHz</span></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">12GB RAM</span></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1TB hard drive x2 (RAID 1)</span></li>
<li style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1GB VRAM (dual 1080p displays)</span></li>
</ul>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">...and this machine is pretty old at this point! Needless to say with 256MB of RAM (double the maximum possible in my Aptiva) and even one emulated CPU Unifix 2.0 barely knows what to do with this new hardware (even if it isn’t real)!</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">While I was never quite sure why I was doing any of this I can tell you that it was very fun to remember how all of this got started. I remember my next distro being RedHat 5.2...</span></b>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com4tag:blogger.com,1999:blog-9220932811142893057.post-63477448866117542132012-06-14T15:57:00.000-04:002012-06-14T16:10:47.273-04:00Everything you wish you didn't need to know about VoIPA few years back I was talking with my editor at <a href="http://oreilly.com/" target="_blank">O'Reilly Media</a> about a book I'd like to write. The book would cover details of the SIP protocol, best practice, interop scenarios, and even a few implementation specifics - FreeSWITCH, Asterisk, OpenSIPS, Kamailio, etc. Basically your typical open source software book only this time it would be SIP protocol inward.<br />
<br />
While my editor liked the idea (I think they have to tell you that) he said there wouldn't be much of a market for it. If I remember correctly his exact words were "Kris, that's a great idea but only 100 people in the world would buy it". Clearly you can't print a physical book through a major publisher with editors, technical reviewers, etc if only 100 people are going to buy it. I tabled the idea.<br />
<br />
Several years later I find myself still regularly talking about SIP and going into many protocol and implementation specifics. Like my editor once told me it seems there aren't a lot of people in this area with either the interest or experience. I guess he was right. Still, SIP is confusing enough and widespread enough that something has to be done.<br />
<br />
Over the past couple of months (off and on - I rarely work uninterrupted these days) I sat down and wrote. Stream of consciousness, without reference, writing. What I ended up with is a (currently) 21 page document I like to call "Everything you wish you didn't need to know about VoIP".<br />
<br />
It's still an early draft but as we say in open source "release early, release often". It has typos. It may not always be factually correct. There are no headings, chapters, or spacing. I may not always use apostrophes correctly. Over time I will correct these mistakes and hopefully (with your help) address other topics of concern or questions my readers may have. I may even divide it into logical chapters at some point! Wow, that would be nice.<br />
<br />
However, as the philosophers say a 100 mile journey begins with a single step. With that said, blog reader, I present to you <a href="https://docs.google.com/document/d/1WBznJ_R-2JgQPbRzFVRvWZdlwIwRrmu3X1J-QmH6Ecs/edit" target="_blank">"Everything you wish you didn't need to know about VoIP"</a>.<br />
<br />
Let me know what you think (I have a comments section and easy to find e-mail address).Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com7tag:blogger.com,1999:blog-9220932811142893057.post-90129723387519401812012-06-04T09:41:00.001-04:002012-06-04T09:41:56.140-04:00Sprechen sie deutsch?Do you speak German?<br />
<br />
I don't. I'm sure this comes as a shock to the many people who have sent me e-mails in German over the years. I suppose my last name may give that impression...<br />
<br />
Anyway, longtime AstLinux community member and contributor Michael Keuter has setup an <a href="http://mksolutions.info/doku.php/Blog" target="_blank">AstLinux focused blog in German</a>. Check it out!Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-55813716105914365162012-03-12T10:23:00.000-04:002012-03-12T10:45:43.165-04:00AstLinux Custom Build Engine now available!Ever since releasing the <a href="http://doc.astlinux.org/devdoc:documentation#development_environment" target="_blank">AstLinux Development Environment</a> several years ago the AstLinux Developers have spent a significant amount of time supporting new users who are (in many cases) building their first image with only minor customizations - slightly different hardware support, different Asterisk versions, etc.<br />
<br />
The trouble is, cross compiling anything is an extremely complicated task. To be honest I'm surprised it works as often as it does. When you step back and really look at what's going on it all starts to seem like magic. Many people working in this space will openly admit to being practitioners of vodoo or one of the other <a href="http://en.wikipedia.org/wiki/Black_magic" target="_blank">dark arts</a>.<br />
<br />
After a brief e-mail exchange last week Lonnie Abelbeck and I decided to do something about this. What if we could host a system with a web interface to build custom AstLinux images for users on demand? What if this system could cache previous image configurations and save them for future users? What if this system could be easily adapted to meet future configuration needs?<br />
<br />
Amazingly, barely a week later, Lonnie has provided all of these features and more. Available immediately, the AstLinux Custom Build Engine is online to build custom AstLinux images that meet your needs.<br />
<br />
In an effort to keep bots, crawlers, and robots in general out we've added simple username and password authentication. The secret is out and the username is "admin" with a password of "astlinux". AstLinux users will recognize these credentials from the default administrative web interface provided with AstLinux. These users will also recognize the familiar tabbed interface.<br />
<br />
Go ahead and <a href="http://build.astlinux.org/" target="_blank">give it a try</a>!<br />
<br />
These interfaces look alike because they share the same DNA. Lonnie Abelbeck has done a great job creating a build system to serve our users now and in the future. Thanks again Lonnie!<br />
<br />
P.S. - Lonnie just found <a href="http://lists.digium.com/pipermail/asterisk-dev/2005-August/014858.html" target="_blank">this post</a> from me, dated 8/25/2005, where I talk about something that looks a lot like build.astlinux.org. If only Lonnie were around back then to help me actually create such a beast!Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-66157405348964236392012-02-13T14:06:00.002-05:002012-02-13T14:06:16.848-05:00Hyperspecialization and the shakeup of a 100 year old industryAs someone who often finds themselves "in the trenches" dealing with some extremely nerdy technical nuances it's often easy to miss the larger picture. I guess Mark Spencer was right when he said "Not many people get excited about telephones but the ones who do get REALLY excited about telephones". <br />
<br />
As someone who's natural inclination is to get stuck in the details I certainly understand this. Some of you might be right there with me. At this point I've gotten so specialized I'm next to useless on some pretty basic "computer things". Think of the aunt or other relative/friend that lights up when they find out you're a "computer guy". Then they inevitably pull you aside at a wedding to ask for help with their printer or "some box that pops up in Windows". I'm happy to not have to feign ignorance any longer: I truly am ignorant on issues like these.<br />
<br />
I primarily use a Mac because it just works - for me. That's not the point I'm trying to make here, though. A Mac works so well for me because I use just two applications: a terminal (<a href="http://www.iterm2.com/" target="_blank">Iterm2</a>, to be exact) and Google Chrome. Ok, ok every once in a while I whip up some crazy Wireshark display syntax but that's another post for another day. For the most part when I need to work I take out my Mac, it comes out of sleep, connects to a network, and I start working. It's a tool.<br />
<br />
As far as anything with a GUI goes that's the extent of my "expertise". If my aunt wanted to ask me about my bash_profile, screenrc settings, IPv4 address exhaustion, or SIP network architecture I may have something to say. Other than that you'll find me speaking in vague generalities than may lead the more paranoid to suspect I'm secretly a double for the CIA or some international crime syndicate member. I wish I were kidding, this has actually happened before although "international crime syndicate" usually gets loosely translated to "drug dealer". How else does a supposed "computer guy" not understand what's wrong with my printer?!?! <br />
<br />
As usual there's a point to all of this. My hyperspecialization, in this case, allows me to forget what is really going on all around me: a shakeup in the 100 year old industry I find myself in and a change in the way we communicate.<br />
<br />
The evolution of the telephone is a strange thing. It is a device and service that has remained largely unchanged for 100 years. I'm not kidding. To this day, in some parts of the United States, the only <a href="http://en.wikipedia.org/wiki/Plain_old_telephone_service" target="_blank">telephone service available</a> could be installed by Alexander Graham Bell himself. Sure there have been many advances since the 1900s but they've been incremental improvements at best - digital services with the same voice bandwidth (dating to <a href="http://en.wikipedia.org/wiki/G.711" target="_blank">1972</a>), various capacity and engineering changes, and of course - the cell phone.<br />
<br />
In the end, however, we're left with a service that isn't much different than what my grandparents had. You still have to phonetically spell various upper-frequency consonants ("S as in Sam, P as in Paul, T as in Tom") because the upper limit of the voice bandwidth on these services is ridiculously low (3.1 kHz). Straining to hear the party at the remote end of a phone has only gotten worse with various digital compression standards in use today - <a href="http://en.wikipedia.org/wiki/Enhanced_Variable_Rate_Codec" target="_blank">EVRC</a>, <a href="http://en.wikipedia.org/wiki/Adaptive_Multi-Rate_audio_codec" target="_blank">AMR</a>, <a href="http://en.wikipedia.org/wiki/G.729" target="_blank">G.729</a>, etc. I love to compare the <a href="http://www.youtube.com/watch?v=HnlqrMWVYCs" target="_blank">"pin drop" Sprint commercials of the 80s and 90s</a> to the Verizon Wireless "CAN YOU HEAR ME NOW?" campaign over 20 years later. We still dial by randomly assigned strings of 10 digit numbers. This is supposedly progress?<br />
<br />
One thing that has changed - the network has gotten bigger. Much bigger. My grandparents may have not had much use for their <a href="http://en.wikipedia.org/wiki/Party_line_%28telephony%29" target="_blank">party line</a> because they didn't have anyone of interest to talk to on the other end. In this manner the network has exploded - and it has exploded using the same standards that have been in place for these past 100 years. I can directly dial a cell phone on the other side of the world and be connected in seconds.<br />
<br />
Meanwhile, there has been another network explosion - IP networks and the internet. The internet, of course, needs no introduction. While I'd love to spend some time talking about IP that's time I don't have at this point. Let's just look at a couple of ways IP has been extremely disruptive for this 100 year old franchise.<br />
<br />
Not many people outside of telecom noticed it at the time but back in 2009 AT&T (THE AT&T) petitioned the FCC to decommission the legacy PSTN (copper and pairs and what-not). Just over two years later we're starting to see some results, and AT&T is <a href="http://vcxc.org/att/" target="_blank">realizing some ancillary benefits</a>.<br />
<br />
As someone who has spent some time (not a lot, thankfully) in these central offices the maze of patch cables, wiring blocks, DC battery banks, etc make you really appreciate the analysis of this report. Normally networks are completely faceless - you go to www.google.com or dial 8005551212 without seeing the equipment that gets you to the other end. The fact that SBC reclaimed as much as 250 MILLION square feet by eliminating this legacy equipment is incredible.<br />
<br />
That's all well and good but what has AT&T done for us, the users? The answer is, unfortunately, both good and bad. AT&T like many physical, trench-digging network providers, has realized they are in the business of providing IP connectivity. They don't have much of a product anymore and the product they do have is becoming more and more of a commodity everyday.<br />
<br />
Getting out of the way is the smartest thing they could be doing. Speaking of AT&T, remember the Apple iPhone deal? At the time a cell phone was a cell phone - AT&T provided an IP network and got some minutes but Apple built an application platform and changed the way people view the devices they carry with them everywhere they go. Huge.<br />
<br />
Watch any sci-fi movie from the past 50 years and one almost ubiquitous "innovation" is the video phone. Did AT&T or some other 100 year old company provide the video phone for baby's first steps to be beamed to Grandma across the country? No - Apple did it with Facetime and a little company from Estonia (Skype) did it over the internet. Thanks to these companies and IP networks we finally have video conferencing (maybe they'll release a 30th anniversary edition of <a href="http://en.wikipedia.org/wiki/Blade_Runner" target="_blank">Blade Runner</a> to celebrate).<br />
<br />
Unfortunately, there will always be people that cling to technologies of days past - this new network works well for all of these applications that were designed for it. Meanwhile, some technologies are being shoehorned in with disastrous results. Has anyone noticed faxing has actually gotten LESS reliable over the past several years? That's what happens when you try to use decades-old modem designs on a completely different network. You might as well try to burn diesel in your gasoline engine.<br />
<br />
The future is the network and the network (regardless of physical access medium) is IP.<br />
<br />
And now, for good measure, here are some random links for further reading:<br />
<br />
<a href="http://www.mgraves.org/" target="_blank">Graves on SOHO Technology</a> - An early advocate of HD Voice, etc.<br />
<a href="http://www.vcxc.org/" target="_blank">The Voice Communication Exchange</a> - Wants to push the world to HD Voice by 2018.<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com1tag:blogger.com,1999:blog-9220932811142893057.post-87685698590009833642011-12-20T17:23:00.000-05:002011-12-20T17:23:06.382-05:00Performance Testing (Part 1)<div style="background-color: transparent;">
<b id="internal-source-marker_0.8060968250501901" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Over the past few years (like many other people in this business) I’ve needed to do performance testing. Open source software is great but this is one place where you need to do your own leg work. This conundrum first presented itself in the Asterisk community. There are literally thousands of variables that can affect system the performance of Asterisk, FreeSWITCH, or any other software solution. In no particular order:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Configuration. Which modules do you have loaded? How are they configured? If you’re using Kamailio, do you do hundreds of huge, slow, nasty DB queries for each call setup? How is your logging configured? Maybe you use Asterisk or FreeSWITCH and so several system calls, DB lookups, LUA scripts, etc? Even the slightest misstep in configuration (synchronous syslogging with Kamailio, for example) can reduce your performance by 90%.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Features in use. Paging groups (unicast) are notorious for destroying performance on standard hardware - every call needs to be setup individually, you need to handle RTP, and some audio mixing is involved. Hardware that can’t do 10 members in a page group using Asterisk or FreeSWITCH may be capable of hundreds of sessions using Kamailio with no media.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Standard performance metrics. “Thousands of calls” you say? How many calls per second? Are you transcoding? Maybe you’re not handling any media at all? What is the delay in call setup?</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- Hardware. This may seem obvious (MORE HERTZ) but even then there are issues... If you’re handling RTP, what are you using for timing? If you have lots of RTP, which network card are you using? Does it and your kernel support MSI or MSI-X for better interrupt handling? Can you load balance IRQs across cores? How efficient (or buggy) is the driver (Realtek I’m looking at you)?!?</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">- The “guano” effect. As features are added to the underlying toolkit (Asterisk, FreeSWITCH, etc) and to your configuration, how is performance affected over time? Add a feature here, and a feature there - and repeat. Over the months and years (even with faster hardware) you may find that each “little” feature reduced call capacity by 5%. Or maybe your calls per second went down by two each time. Not a big deal overall yet over time this adds up - assuming no other optimizations your call capacity could be down by 50% after ten “minor” changes. It adds up - it really does.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even when pointing out all of these issues you’d still be surprised how often one is faced with the question “Well yeah but how many calls can I handle on my dual core Dell server?”.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In almost every case the best answer is “Get your hardware, develop your solution, run </span><a href="http://sipp.sourceforge.net/"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">sipp</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> against it and see what happens”. That’s really about as good as we can do.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SIPP is a great example of a typical, high quality open source tool. In true “Unix philosophy” it does one thing and it does it well: SIP performance testing. SIPP can be configured to initiate (or receive) just about any conceivable SIP scenario - from simple INVITE call handling to full </span><a href="http://en.wikipedia.org/wiki/SIMPLE"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">SIMPLE</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> test cases. In these tests SIPP will tell you call setup time, messages received, successful dialogs, etc.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">SIPP even goes a step further and includes some support for RTP. SIPP has the ability to echo RTP from the remote end or even replay RTP from a PCAP file you have saved to disk. This is where SIPP starts to show some deficiencies. Again, you can’t blame SIPP because SIPP is a SIP performance testing tool - it does that and it does it well. RTP testing leaves a lot to be desired. First of all, you’re on your own when it comes to manipulating any of the PCAP parameters. Length, content, codec, payload types, etc, etc need to be configured separately. This isn’t a problem, necesarily, as there are various open source tools to help you with some of these tasks. I won’t get into all of them here but they too leave something to be desired.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What about analyzing the quality of the RTP streams? SIPP provides mechanisms to measure various SIP “quality” metrics - SIP response times, SIP retransmits, etc. With RTP you’re on your own. Once again, sure, you could setup tshark on a SPAN port (or something) to do RTP stream analysis on every stream but this would be tedious and (once again) subject you to some of the harsh realities of processing a tremendous amount of small packets in software on commodity hardware.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Let’s face it - for a typical B2BUA handling RTP the numbers add up very quickly - let’s assume 20ms packetization for the following:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Single RTP stream = 50 packets per second (pps)</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Bi-directional RTP stream = 100 pps</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A-leg bi-directional RTP stream = 100 pps</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">B-leg bi-directional RTP stream = 100 pps</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A leg + B leg = 200 pps PER CALL</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What does this look like with 10,000 channels (using g711u)?</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">952 mbit/s (close to Gigabit wire speed) in each direction</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1,000,000 (total) packets per second</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Open source software is great - it provides us with the tools to (ultimately) build services and businesses. Many of us choose what to focus on (our core competency). At Star2Star we provide business grade communication services and we spend a lot of time and energy to build these services because it’s what we do. We don’t sell, manufacture, or support testing platforms.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At this point some of you may be getting an idea... Why don’t I build/design an open source testing solution? It’s a good question and while I don’t want to crush your dreams there are some harsh realities:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1) This gets insanely complicated, quickly. Anyone who follows this blog knows SIP itself is complicated enough.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2) Scaling becomes a concern (as noted above).</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3) Who would use it?</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The last question is probably the most serious - who really needs the ability to initiate 10,000 SIP channels at 100 calls per second while monitoring RTP stream quality, etc? SIP carriers? SIP equipment manufacturers? A few SIP software developers? How large is the market? What kind of investment would be required to even get the project off the ground? What does the competition look like? While I don’t have the answers to most of these questions I can answer the last one.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Commercial SIP testing equipment is available from a few vendors:</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><a href="http://www.spirent.com/Solutions-Directory/Abacus.aspx"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Spirent</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><a href="http://www.empirix.com/products/hammer_g5.asp"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Empirix</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><a href="http://www.ixiacom.com/products/display?skey=ixload_sip"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Ixia</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">...and I’m sure others. We evaluated a few of these solutions and I’ll be talking more about them in a follow-up post in the near future.</span></b></div>
<div style="background-color: transparent;">
<b id="internal-source-marker_0.8060968250501901" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></b></div>
<div style="background-color: transparent;">
<b id="internal-source-marker_0.8060968250501901" style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Stay tuned because this series is going to be good! </span></b></div>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-70153255121694867092011-12-02T09:58:00.001-05:002011-12-02T10:03:22.514-05:00Star2Star Gets NoticedJust a quick one today (and some shameless self promotion on my part)... Star2Star has been recognized on a few "lists" this year, check it out:<br />
<br />
<a href="http://www.inc.com/inc5000/profile/star2star-communications" target="_blank">Inc 500 </a><br />
<a href="http://www.forbes.com/lists/2011/28/most-promising-companies-11_Star2Star-Communications_OL02.html" target="_blank">Forbes 100 "Most Promising"</a> <br />
<br />
I'm lucky enough to tell people the same story all of the time - when I was a little kid I played with all of this stuff because I thought it was fun and I loved it. Only later did I realize that one day I'd be getting paid for it. I certainly never thought it could come to this!<br />
<br />
Ok, enough of that for now. I'll be getting back to some tech stuff soon...Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com1tag:blogger.com,1999:blog-9220932811142893057.post-63943062074084278062011-11-15T14:45:00.001-05:002011-11-15T14:47:05.876-05:00Building a Startup (the right way)<div style="background-color: transparent;">
<span id="internal-source-marker_0.2732291151769459" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(Continued from <a href="http://blog.krisk.org/2011/11/bulding-startup.html">Building a Startup</a>) </span></div>
<div style="background-color: transparent;">
<span id="internal-source-marker_0.2732291151769459" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<div style="background-color: transparent;">
<span id="internal-source-marker_0.2732291151769459" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Our way wasn’t working. To put it mildly our “business grade” solution didn’t perform much better than Vonage. We became to exemplify VoIP - jittery calls, dropped calls, one way calls, etc, etc, etc. Most of this was because of the lack of quality ITSPs at that time. Either way our customers didn’t care. It was us. If we went to market with what we had the first time around we were going to loose.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The problem was the other predominant architecture at the time was “hosted”. Someone hosts a PBX for you and ships you some phones. You plug them in behind your router and magically you have a phone system. They weren’t doing much better. Sure, their sales looked good but even then it was becoming obvious customer churn was quite high. People didn’t like hosted either, and for good reason. Typically they have less control over the call than we do.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As I’ve eluded before I thought there was a better way. We needed to host the voice applications where it made the most “sense”. We were primarily using Asterisk and with a little creative provisioning, a kick-ass SIP proxy, and enough Asterisk machines we could build the perfect business PBX - even if that meant virtually none of it existed at the customer premise. Or maybe all of it did. That flexibility was key. After a lot of discussions, whiteboard sessions, and late nights everyone agreed. We needed a do-over.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So we got to work and slowly our new architecture began to take shape. We added a kick-ass SIP proxy (OpenSER). OpenSER would power the core routing between various Asterisk servers each meeting different needs - IVR/Auto Attendant, Conferencing, Voicemail, remote phones (for “hosted” phones/softphones), etc. The beauty was the SIP proxy could route between all of these different systems including the original AstLinux system at the customer premise. Customer needs to call voicemail? No problem - the AstLinux system at the CPE fires an INVITE off to the proxy and the proxy figures out where their voicemail server is. The call is connected and the media goes directly between the two endpoints. Same thing for calls between any two points on the network - AstLinux CPE to AstLinux CPE, PSTN to voicemail, IVR to conference.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is a good time to take a break and acknowledge what really made this all possible - OpenSER. While it’s difficult to explain the exact history and family tree with any piece of SER software I can tell you one thing - this company would not be possible without it. There is no question in my mind. It’s now 2011 and whether you select </span><a href="http://www.kamailio.org/"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Kamailio</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> or </span><a href="http://opensips.org/"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">OpenSIPS</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> for your SIP project you will not be sorry. Even after five years you will not find a more capable, flexible, scalable piece of SIP server software. It was one of the best decisions we ever made.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Need to add another server to meet demand for IVR? No problem, bring another server online, add the IP to a table and presto - you’re now taking calls on your new IVR. Eventually a new IVR lead to several new IVRs, voicemail servers, conference systems, web portals, mail servers, various monitoring systems, etc.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What about infrastructure? Starting at our small scale, regional footprint, and focus on quality we began to buy our own PRIs and running them on a couple of Cisco AS5350XM gateways. This got us past our initial issues with questionable ITSPs. Bandwidth was becoming another problem... We had an excellent colocation provider that provided blended bandwidth but still we needed more control. Here came BGP, ARIN, AS numbers, a pair of Cisco 7206VXRs w/ G2s, iBGP, multiple upstream providers, etc.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At times I would wonder - whatever happened to spending my time worrying about cross compilers? Looking back I’m not sure which was worse - GNU autoconf cross-compiling hell or SIP interop, BGP, etc. It’s fairly safe to say I’m a sadomasochist either way.</span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even with all of the pain, missteps, and work we finally had an architecture to take to market. It would be the architecture that would serve us well for several years. Of course there was more work to be done...</span></div>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com5tag:blogger.com,1999:blog-9220932811142893057.post-31701277105089333862011-11-02T16:49:00.000-04:002011-11-15T15:04:46.794-05:00Building a Startup<div style="background-color: transparent;">
<span id="internal-source-marker_0.025660596787929535" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(Continued from <a href="http://blog.krisk.org/2011/10/starting-startup.html">Starting a Startup</a>) </span></div>
<div style="background-color: transparent;">
</div>
<div style="background-color: transparent;">
<span id="internal-source-marker_0.025660596787929535" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After several days of meetings in Sarasota we determined:</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1) I was moving to Sarasota to start a company with Norm and Joe.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2) We were going to utilize open source software wherever possible (including AstLinux, obviously).</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3) The Internet was the only ubiquitous, high quality network to build a nationwide platform.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4) The Internet was only getting more ubiquitous, more reliable, and faster in coming months/years/decades/etc.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5) We were going to take advantage of as much of this as possible.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">These were some pretty lofty goals. Remember, this is early 2006. Gmail was still invitation-only beta. Google docs didn’t exist. Amazon EC2 didn’t exist. “Cloud computing” hadn’t come back into fashion yet. The term itself didn’t exist. The Internet was considered (by many) to be “best effort”, “inherently unreliable”, and “unsuitable” for critical communications (such as real time business telephony). There were many naysayers who were confident this would be a miserable failure. As it turns out, they were almost right.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We thought the “secret sauce” to business grade voice over the internet was monitoring and management. If one could monitor and manage the internet connection business grade voice should be possible. Of course this is very ambiguous but it lead to several great hires. We hired </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Joe had already deployed several embedded Asterisk systems to various businesses in the Sarasota area. They used an embedded version of Linux he patched together and a third party (unnamed) “carrier” to connect to the PSTN. The first step was upgrading these machines and getting them on AstLinux. Once this was accomplished we felt confident enough to proceed with our plan. This was Star2Star Communications and in the beginning of 2006 it looked something like this:</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1) Soekris net4801 machines running AstLinux on the customer premise.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2) Grandstream GXP-2000 phones at each desk.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3) Connectivity to a third party “ITSP”.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4) Management/monitoring systems (check IP connectivity, phone availability, ITSP reliability, local LAN, etc).</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">5) Central provisioning of AstLinux systems, phones, etc.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This was Star2Star and there was something I really liked about it - it was simple. Anyone who knows me or knows of my projects (AstLinux, for example) has to know I favor simplicity whenever possible. Keep it simple, keep it simple, keep it simple (stupid).</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As time went on we started to learn that maybe this was too simple. We didn’t have enough control. Out monitoring wasn’t as mature as it should be. We didn’t pick the right IP phones. These could be easily fixed. However, we soon realized our biggest mistake was architecture (or lack thereof). This wasn’t going to be an easy fix.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">We couldn’t find an ITSP that offered a level of quality we considered to be acceptable. Very few ITSPs had any more experience with VoIP, SIP, and the internet than we did. More disturbing, however, was an almost complete lack of focus on quality and reliability. No process.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What we (quickly) discovered is the extremely low barrier to entry for ITSPs, especially back then. Virtually anyone could install Asterisk on a $100/mo box in a colo somewhere, buy dialtone from someone (who knows) and call themselves an ITSP. After going through several of these we discovered we needed to do it ourselves.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even assuming we could solve the PSTN connectivity problem we discovered yet another issue. All of the monitoring and management in the world cannot make up for a terrible last mile. If the copper in the ground is rotting and the DSL modem can only negotiate 128kbps/128kbps that’s all you’re going to get. To make matters worse in the event of a cut or outage the customer would be down completely. While that may have always happened with the PSTN and an on premise PBX we considered this to be unacceptable.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So then, in the eleventh hour, just before launch I met with the original founders and posed a radical idea - scrap almost everything. There was a better way.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(Continued in <a href="http://blog.krisk.org/2011/11/building-startup-right-way.html">Building a Startup (the right way)</a>) </span></div>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-17804940268341269012011-10-25T14:09:00.000-04:002011-11-02T16:50:04.492-04:00Starting a Startup<span class="Apple-style-span" style="color: black; font-family: Times; font-size: small; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;"></span><br />
<div style="background-color: transparent;">
<span id="internal-source-marker_0.45115007110871375" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I know I’ve apologized for being quiet in the past. This is not one of those times because (as you’ll soon find out) I’ve been hard at work and only now can I finally talk about it.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Six years ago I was spending most of my time working with Asterisk and AstLinux. I spent a lot of time promoting both - working the conference circuit, blogging, magazines, books, etc. Conferences are a great way to network and meet new people. I did just that. With each conference I attended came new business opportunities. Sure, not all of them were a slam dunk and eventually I started to pick and chose which conferences I considered worthy of the time and investment.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">For anyone involved with Asterisk Astricon is certainly worthy of your time and energy - the mecca of the Asterisk community. Astricon was always a whirlwind and 2005 was no exception. We were in Anaheim, California and embedded Asterisk was starting to really heat up. I announced my port of </span><a href="http://www.astlinux.org/"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">AstLinux</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> to </span><a href="http://www.gumstix.com/"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Gumstix</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and announced the “World’s Smallest PBX”, leading to an interview and </span><a href="http://www.linuxfordevices.com/c/a/News/Worlds-smallest-VoIP-PBX/"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">story</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in LinuxDevices. I worked a free community booth (thanks Astricon) with </span><a href="http://the-edge.blogspot.com/"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Dave Taht</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and was introduced to </span><a href="http://en.wikipedia.org/wiki/John_Draper"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Captain Crunch</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> (that’s another post for another day).</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It was at Astricon in 2005 that I also met one of my soon to be business partners (although I certainly didn’t know it at the time). While I was promoting embedded Asterisk and AstLinux I met a man from Florida named Joe Rhem. Joe had come up with the idea of using embedded Asterisk systems as the cornerstone of a new way to provide business grade telephone services. Joe and I met for a few minutes and discussed the merits of embedded Asterisk. Unfortunately (and everyone already knows this) I don’t remember meeting with Joe. Like I said Astricon was always a whirlwind and I had these conversations with dozens if not hundreds of people at each show. I made my way through Astricon, made a pit stop in Santa Clara for (the now defunct) </span><a href="http://blog.isp-planet.com/blog/2009/02/ispcon-closes.html"><span style="background-color: transparent; color: #000099; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ISPCon</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> and then returned home to Lake Geneva, WI with a stack of business cards, a few new stories, and a lot of work to finish (or start, depending on your perspective).</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A couple of months later I received an e-mail from Joe Rhem discussing how he’d like to move forward with what we discussed in Anaheim. Joe had recruited another partner to lead the new venture. Norm Worthington was a successful serial entrepreneur and his offer to lead the company was the equivalent of “having General Patton lead your war effort”. After some catch up I was intrigued with Joe’s idea. A few hours on the phone later everyone was pretty comfortable with how this could work.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Now I just needed to fly to Sarasota, FL (where’s that - sounds nice, I thought) to meet with everyone, discuss terms, plan a relocation, and (most importantly) start putting the company, product, and technology together.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">A short time later I found myself arriving in Sarasota. It was early January and I coming from Wisconsin I couldn’t believe how nice it was. Looking back on it I’m sure Norm and Joe were very confident I’d be joining them in Sarasota. Working with technology I love “in paradise”, how could I resist?</span><br />
<br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">(Continued in <a href="http://blog.krisk.org/2011/11/bulding-startup.html">Building a Startup</a>) </span></div>Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-27264606831209871502010-10-26T17:39:00.005-04:002010-10-26T18:47:46.400-04:00Breaking RFC compliance to improve monitoringA colleague came to me today and had a troubling issue. He's using <a href="http://sipsak.org/">sipsak</a> and <a href="http://www.nagios.org/">nagios</a> to monitor some SIP endpoints. Pretty standard so far, right? He noticed that when using UDP and checking on an endpoint that was completely offline sipsak would take over 30 seconds to finally return with an error. Meanwhile Nagios would block and wait for sipsak to return...<br /><br />Without a simple command line option in sipsak that appeared to change this behavior, we had to enter the semi-complicated world of SIP timers. I feared that to change this behavior we'd have to do some things that might not necessarily be RFC compliant...<br /><br />What's this? For once I'm actually suggesting you do something against the better advice of an RFC?<br /><br />That's right, I am.<br /><br /><a href="http://www.ietf.org/rfc/rfc3261.txt">RFC3261</a> defines multiple timers and timeouts for messages and transactions. It says things like:<br /><br />"If there is no final response for the original request in 64*T1 seconds"<br /><br />"The UAC core considers the INVITE transaction completed 64*T1 seconds after the reception of the first 2xx response."<br /><br />"The 2xx response is passed to the transport with an interval that starts at T1 seconds and doubles for each retransmission until it reaches T2 seconds"<br /><br />Without even knowing what "T1" is you can start to see that it's a pretty important timing parameter and (more or less) serves as the father of all timeouts in SIP. Let's look at section 17 to find out what T1 is:<br /><br />"The default value for T1 is 500 ms. T1 is an estimate of the RTT between the client and server transactions. Elements MAY (though it is NOT RECOMMENDED) use smaller values of T1 within closed, private networks that do not permit general Internet connection. T1 MAY be chosen larger, and this is RECOMMENDED if it is known in advance (such as on high latency access links) that the RTT is larger. Whatever the value of T1, the exponential backoffs on retransmissions described in this section MUST be used."<br /><br />T1 is essentially a variable for RTT between two endpoints that serves as a multiplier for other timeouts. Unless we know better T1 should default to 500ms, which is quite high. Some implementations (such as Asterisk with the SIP peer qualify option) automatically send OPTIONS requests to endpoints in an effort to better determine RTT instead of using the RFC default of 500ms.<br /><br />In reading through the sipsak source code it appeared to be RFC compliant for timing, using a default T1 value of 500ms and a transaction timeout value of 64*T1. This is why it was taking over 30 seconds (32 seconds to be exact) for sipsak to finally timeout and return the status code to nagios. This comes directly from the RFC:<br /><br />"For any transport, the client transaction MUST start timer B with a value of 64*T1 seconds (Timer B controls transaction timeouts)."<br /><br />This is all well and good but what happens when you don't have a way to dynamically determine T1 and you can't wait T1*64 (32s) for your results like my sipsak/nagios check earlier? Simple: you go renegade, throw out the RFC, and hack the sipsak source yourself!<br /><br />So I had three options:<br /><br />1) Change the default value of T1.<br />2) Change the value of T2 by changing the multiplier or setting a static timeout.<br />3) Some combination of both.<br /><br />I decided to go with option #3 (RFC be damned). Why?<br /><br />1) 500ms is crazy high for most of our endpoints. At a glance 100ms would be fine for ~90% of them. I'll pick 150ms.<br />2) I don't need that many retransmits. If the latency and/or packet loss is that bad I'm not going to wait (my RTP certainly isn't) and I just want to know about it that much quicker.<br /><br />So I ended up with a quick easy patch to sipsak:<br /><br />diff -urN sipsak-0.9.6.orig/sipsak.h sipsak-0.9.6/sipsak.h<br />--- sipsak-0.9.6.orig/sipsak.h 2006-01-28 16:11:50.000000000 -0500<br />+++ sipsak-0.9.6/sipsak.h 2010-10-26 18:38:45.000000000 -0400<br />@@ -102,11 +102,7 @@<br /> # define FQDN_SIZE 100<br /> #endif<br /><br />-#ifdef HAVE_CONFIG_H<br />-# define SIP_T1 DEFAULT_TIMEOUT<br />-#else<br />-# define SIP_T1 500<br />-#endif<br />+#define SIP_T1 150<br /><br /> #define SIP_T2 8*SIP_T1<br /><br />diff -urN sipsak-0.9.6.orig/transport.c sipsak-0.9.6/transport.c<br />--- sipsak-0.9.6.orig/transport.c 2006-01-28 16:11:34.000000000 -0500<br />+++ sipsak-0.9.6/transport.c 2010-10-26 18:38:51.000000000 -0400<br />@@ -286,7 +286,7 @@<br /> }<br /> }<br /> senddiff = deltaT(&(srt->starttime), &(srt->recvtime));<br />- if (senddiff > (float)64 * (float)SIP_T1) {<br />+ if (senddiff > inv_final) {<br /> if (timing == 0) {<br /> if (verbose>0)<br /> printf("*** giving up, no final response after %.3f ms\n", senddiff);<br /><br />This changes the value of T1 to 150ms (more reasonable for most networks) and allows you to specify the number of retransmits (and thus the total timeout) using -D on the sipsak command line:<br /><br />kkmac:sipsak-0.9.6-build kris$ ./sipsak -p 10.16.0.3 -s sip:ext_callqual@asterisk -D1 -v<br />** timeout after 150 ms**<br />*** giving up, no final response after 150.334 ms<br /><br />kkmac:sipsak-0.9.6-build kris$ ./sipsak -p 10.16.0.3 -s sip:ext_callqual@asterisk -D2 -v<br />** timeout after 150 ms**<br />** timeout after 300 ms**<br />*** giving up, no final response after 460.612 ms<br /><br />kkmac:sipsak-0.9.6-build kris$ ./sipsak -p 10.16.0.3 -s sip:ext_callqual@asterisk -D4 -v<br />** timeout after 150 ms**<br />** timeout after 300 ms**<br />** timeout after 600 ms**<br />*** giving up, no final response after 1071.137 ms<br /><br />kkmac:sipsak-0.9.6-build kris$<br /><br />Needless to say our monitoring situation is much improved.Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-6231471004808065862010-08-05T11:56:00.004-04:002010-08-05T12:03:56.621-04:00A ClueCon UpdateCluecon is going very well this year... I spoke the first day and have spent the rest of my time here enjoying the presentations and interacting with the community.<br /><br />A few highlights:<br /><br /><ul><li>Perfect wireless provided by <a href="http://meraki.com/">Meraki</a>. I've never been to a tech conference where the wifi has kept up with the crowd. Well done.</li><li>The Trump Tower. Phenomenal.</li><li>FreeSWITCH HA support in Sofia! This is worthy of its own post and it will have one when I get back and play with it. In the meantime my guy Jay Binks has been working to <a href="http://wiki.freeswitch.org/wiki/Freeswitch_HA">document</a> this exciting new feature.<br /></li><li>Chicago. I just LOVE this town.</li></ul>More later... I've got to get back to the conference!Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0tag:blogger.com,1999:blog-9220932811142893057.post-41436890265106320862010-05-21T14:54:00.009-04:002010-07-28T15:47:01.831-04:00A ClueCon Preview...A while back I saw a preview for the new A-Team movie. While the movie itself looks horrible I was reminded of the original TV series with its many interesting characters and catch phrases. Among my personal favorites?<br /><br />I love it when a plan comes together.<br /><br />That's exactly how I feel with one of my "pet projects" from the past couple of months. Much like Hanibel and the A-Team I was up against formidable issues in trying to accomplish my task: implementing a flexible (very flexible), reasonably high performance LCR server that could be added to my existing architecture.<br /><br />First I needed to select an LCR "engine". Multiple possibilities were considered but I left the final recommendation up to the DB and billing teams I work with. They selected mod_lcr from FreeSWITCH. While I was certain droute from OpenSIPS (or something similar) would have higher performance I accepted their recommendation. After playing with mod_lcr a bit I can also see its potential.<br /><br />So now the question was: can FreeSWITCH respond with the proper SIP signaling (300 Multiple Choices)? Using the redirect application from mod_dptools it could not. I created a bounty to add multiple Contact/300 Multiple Choices functionality to FreeSWITCH. Tony had it implemented that day.<br /><br />With the ability to respond properly I now had to get the data. Mod_lcr looked nice but it certainly wasn't designed for this application. All of the default syntax, tables, etc showed it being used with FreeSWITCH for FreeSWITCH. The tables and code used several bridge specific syntax examples. I hacked mod_lcr to return data to mod_dptools/redirect properly. A created a JIRA issue with my patch and a couple of days later Rupa had it committed.<br /><br />So now FreeSWITCH could be a route server. All I needed to do was make sure OpenSIPS could route from what FreeSWITCH returned. Turns out it could not. RFC 3261 (section 21.3.1) states "...the SIP response MAY contain several Contact fields or a list of addresses in a Contact field." The Sofia stack from FreeSWITCH used multiple Contact headers, each with its own URI. OpenSIPS would only parse the first one returned. Sofia couldn't be changed easily so OpenSIPS would need to be changed (it was non-compliant anyway). Without this change there is no ability to handle multiple contacts and only the first would be used. It could be worse but obviously this wasn't good enough.<br /><br />I contacted Bogdan from OpenSIPS to see what it would take to update the parser to handle multiple Contact headers. He indicated it would take four hours or so. Once he got back to me I had an OpenSIPS system that would handle multiple contact headers and create new branches from a failure route as desired.<br /><br />So how did it all turn out? Well, you have two ways to hear the end of this story:<br /><br />1) Attend ClueCon at the Trump Hotel in Chicago, IL in early August.<br />2) Wait until mid-August for an update here.<br /><br />I'll make sure to post all of my materials - conference presentation, sipp scenarios for testing, OpenSIPS configuration, FreeSWITCH configuration, DB tweaks, etc.<br /><br />Too late to make it to ClueCon this year? Just make sure to register next year, I'm sure I'll be there.Anonymoushttp://www.blogger.com/profile/06405875458561185080noreply@blogger.com0