After my original post on “packets of death” I’ve spent the last couple of weeks receiving reports from users all over the world ranging from:
- Able to reproduce on 82574L with your “packets of death”
- Not able to reproduce on 82574L or any other controller
-
Not able to reproduce with your “packets of death” but experiencing
identical, sporadic failures across a wide range of controllers
Of
course the last category intrigued me the most. There seem to be an
awful lot of people experiencing sporadic failures of their ethernet
controllers but many of them (as I noted in another update)
don’t have the time or tools to diagnose the issue further. In most
cases the symptoms are identical to what I described in my original
post:
- Ethernet controller loses link (or reports some other hardware error)
- Varying amounts of time since boot (hours, days, weeks)
- Can only be resolved by a reboot or in some cases a complete power cycle
Many
of these users have been dealing with these failures in various ways
but have been unable to find a root cause. I’ve created a tool to help
them with their diagnosis. It’s called findpod and it’s been tested on various Debian-based Linux distributions. Findpod uses the excellent ifplugd daemon, the venerable tcpdump, and screen.
Once
installed and started iflpugd will patiently wait to receive link
status notifications from the Linux kernel. Once link is detected on
the target interface it will start a tcpdump session running inside of
screen. This tcpdump session will log all packets sent and received on
that interface. Here’s the thing - many of these failures are reported
after days or weeks of processed traffic - the tcpdump capture file
could easily reach several gigabytes or more! Here’s where one key
trick in findpod comes into play - by default findpod will only log the
last 100MB sent or received on the target interface. As long as ifplugd
doesn’t report any link failures tcpdump will keep writing to the same
100MB circular capture file.
What
happens when the interface loses link? This is the second unique feature of findpod. When ifplugd reports a loss of link it will wait for 30
seconds before stopping the packet capture and moving the capture file
to a meaningful (and known) name. If you think your ethernet controller
failures could be related to the types of traffic you’re sending or
receiving (as I discovered with my “packets of death”) findpod will help
you narrow it down to (at most) 100MB of network traffic, even if the
capture runs for weeks and your interface handles GBs of data!
Of
course (more than likely) it’s even easier than that; if your link loss
is being caused by a specific received packet it will be the last
packet in the capture file provided by findpod and you’ll only have a
100MB capture file to work with. If your issue is anything like mine you should be able to isolate it down to a specific packet that you can feed to tcpreplay; reproducing your controller issue on demand.
Please tell me about your experiences with findpod. As always, comments and suggestions are welcome!
Wednesday, February 20, 2013
Friday, February 8, 2013
Packets of Death - UPDATE
UPDATE - Intel has pointed me towards the successor to the 82574, which includes some of the features I suggest here. I think it's safe to say we're all looking forward to this chip hitting the streets!
The last 48 hours has been interesting, to say the least.
My original post gathered much more attention than I originally thought. I’ll always remember being on a conference call and having someone tell me “Hey, you’re on Slashdot”. Considering the subject matter I suppose I should have expected that.
As of today, here’s what I know:
Many of you have shared the results of your testing. The vast majority of tested Intel ethernet controllers do not appear to be affected by this issue.
Intel has responded with an expanded technical explanation of the issue. I also received a very pleasant and professional phone call from Douglas Boom (at Intel) to update me on their assessment of the situation and discuss any ongoing concerns or problems I may have. Thank you Doug and well done Intel! Note to other massive corporations that could be presented with issues like this: do what Intel did.
To summarize their response, Intel says that a misconfigured EEPROM caused this issue. The EEPROM is written by the motherboard manufacturer and not Intel. Intel says my motherboard manufacturer did not follow their published guidelines for this process. Based on what I’ve seen, how my issue was fixed, and what I’m learning from the crowdsourced testing process this seems like a perfectly plausible explanation. Once again, thanks Intel!
However, I still don’t believe this issue is completely isolated to this specific instance and one motherboard manufacturer. For one, I have received at least two confirmed reports from people who were able to reproduce this issue - my “packet of death” shutting down 82574L hardware from different motherboard manufacturers. This doesn’t surprise me at all.
One thing we’re reminded of in this situation is just how complex all of these digital systems have become. We’re a long way from configuring ethernet adapters with IRQ jumpers. Intel has designed an incredibly complex ethernet controller - the datasheet is 490 pages long! Of course I’m not faulting them for this - the features available in this controller (or many other controllers) dictate this level of complexity - management/IPMI features, WOL, various offloading mechanisms, interrupt queues, and more. This complexity doesn’t even scratch the surface of the various other systems involved in getting data across the internet and into your eyeballs!
Like any sufficiently advanced product all of these features are driven by a configuration mechanism. The Linux kernel module for the 82574L (e1000e) has various options that can be passed to modify the behavior of the adapter. Makes sense. If I passed some stupid or unknown parameter to this module I would expect it to return with some kind of error informing me of my mistake. I’m only human, mistakes are going to happen.
At a lower level Intel has exposed these EEPROM configuration parameters to control various aspects of the controller. As Intel says these EEPROM values are to be set by the motherboard manufacturer. Here’s where the problem lies - it’s certainly possible this could be done incorrectly. Motherboard manufacturers are human and they make mistakes too.
Unfortunately, as we’ve learned in this case, there isn’t quite the same level of feedback when EEPROM misconfigurations happen. In my previous example if I pass unknown parameters to a kernel module it’s going to come back and say “Hey - I don’t know what that is (dummy)” and exit with an error.
As I’ve shown in some cases (mine) if an EEPROM is misconfigured everything appears normal until some insanely specific packet is received. Then the controller shuts down, for some reason.
Does that behavior make sense to anyone?
I suggest the following:
1) Make future controllers have as much in-hardware sane behavior as possible when unknown conditions are encountered. Error checking, basically. Users can input data on a web form, that’s why there’s error checking. Everyone knows users do stupid things. Clearly some of the people programming Intel EEPROMs for motherboard OEMs do stupid things too. What is sane default behavior? EEPROM error encountered = adapter shutdown and error message. Give the user notification and provide some mechanism for EEPROM validation and management...
2) Put more EEPROM validation in operating system drivers. Intel maintains ethernet drivers for various platforms. Why aren’t these drivers doing more validation of the adapter EEPROM? If my EEPROM was so badly misconfigured, why couldn’t the e1000e module have discovered that and notified me?
3) Produce and support an external tool for EEPROM testing, programming, and updating. In the course of working with Intel last fall I was provided a version of this tool for my testing so I know it exists. While I can understand why you don’t want random users messing with their EEPROM (and causing potential support nightmares) it seems the benefits would clearly outweigh any potential problems (of which there are already plenty).
The reality is Intel has no idea how many systems are affected by this issue or could be affected by issues like it. How could they? They’re expecting motherboard OEMs to follow their instructions (and understandably so). Just look at the combination of variables required to reproduce this issue:
- Intel 82574L
- Various specific misconfigured bytes in the EEPROM
- An insanely specific packet with the right value at just the right byte, received at a specific time
While most people weren’t able to reproduce this issue with their controller and EEPROM combinations it did kick off various discussions of periodic, random, sporadic failures across a wide range of ethernet adapters and general computing weirdness. A quick Google search returns a wide assortment of complaints with these adapters (and others like it) from a whole slew of users. EEPROM corruption. Random adapter resets. Packet drops. Various latency issues. PCI bus scan errors. ASPM problems. The list goes on and on.
Perhaps the “packet of death” for a slightly misconfigured Intel 82579 (for example) is my packet shifted 20 bytes in one direction or the other. Who knows? Please, please, please Intel - lets do everyone a favor and get these EEPROMs under control. End users update firmware all of the time - routers, set-top boxes, sometimes even their cars! Why can’t we have some utility to make sure our ethernet adapters aren’t just waiting to freak out when they receive the wrong packet?
I don’t believe in magic, swamp gas, sun spots, or any of the other “explanations” offered for some of the random strange behavior we often see with these complex devices (ethernet adapters or otherwise). That’s why I spent so long working on this issue to find a root cause (well that and screaming customers). I, like anyone else, encounter bugs and general weirdness in devices and software everyday in my life. Most of the time how do I respond to these bugs? I reboot, shrug my shoulders, say “that was weird”, and move on. Meanwhile I know, deep down, that there is a valid explanation for what just happened. Just like there was with my ethernet controllers.
Even with the explanation offered by Intel we could go much deeper. Why these bytes at that location? Why this packet? What’s up with the “inoculation” effect of some of the values? There are still many unanswered questions.
I’ve enjoyed reading many others report their tales of “extreme debugging” with the digital devices in their lives. It seems I’m not the only one that isn’t always satisfied with saying “that was weird” and moving on.
I've said it before and I'll say it again - I love the internet!
The last 48 hours has been interesting, to say the least.
My original post gathered much more attention than I originally thought. I’ll always remember being on a conference call and having someone tell me “Hey, you’re on Slashdot”. Considering the subject matter I suppose I should have expected that.
As of today, here’s what I know:
Many of you have shared the results of your testing. The vast majority of tested Intel ethernet controllers do not appear to be affected by this issue.
Intel has responded with an expanded technical explanation of the issue. I also received a very pleasant and professional phone call from Douglas Boom (at Intel) to update me on their assessment of the situation and discuss any ongoing concerns or problems I may have. Thank you Doug and well done Intel! Note to other massive corporations that could be presented with issues like this: do what Intel did.
To summarize their response, Intel says that a misconfigured EEPROM caused this issue. The EEPROM is written by the motherboard manufacturer and not Intel. Intel says my motherboard manufacturer did not follow their published guidelines for this process. Based on what I’ve seen, how my issue was fixed, and what I’m learning from the crowdsourced testing process this seems like a perfectly plausible explanation. Once again, thanks Intel!
However, I still don’t believe this issue is completely isolated to this specific instance and one motherboard manufacturer. For one, I have received at least two confirmed reports from people who were able to reproduce this issue - my “packet of death” shutting down 82574L hardware from different motherboard manufacturers. This doesn’t surprise me at all.
One thing we’re reminded of in this situation is just how complex all of these digital systems have become. We’re a long way from configuring ethernet adapters with IRQ jumpers. Intel has designed an incredibly complex ethernet controller - the datasheet is 490 pages long! Of course I’m not faulting them for this - the features available in this controller (or many other controllers) dictate this level of complexity - management/IPMI features, WOL, various offloading mechanisms, interrupt queues, and more. This complexity doesn’t even scratch the surface of the various other systems involved in getting data across the internet and into your eyeballs!
Like any sufficiently advanced product all of these features are driven by a configuration mechanism. The Linux kernel module for the 82574L (e1000e) has various options that can be passed to modify the behavior of the adapter. Makes sense. If I passed some stupid or unknown parameter to this module I would expect it to return with some kind of error informing me of my mistake. I’m only human, mistakes are going to happen.
At a lower level Intel has exposed these EEPROM configuration parameters to control various aspects of the controller. As Intel says these EEPROM values are to be set by the motherboard manufacturer. Here’s where the problem lies - it’s certainly possible this could be done incorrectly. Motherboard manufacturers are human and they make mistakes too.
Unfortunately, as we’ve learned in this case, there isn’t quite the same level of feedback when EEPROM misconfigurations happen. In my previous example if I pass unknown parameters to a kernel module it’s going to come back and say “Hey - I don’t know what that is (dummy)” and exit with an error.
As I’ve shown in some cases (mine) if an EEPROM is misconfigured everything appears normal until some insanely specific packet is received. Then the controller shuts down, for some reason.
Does that behavior make sense to anyone?
I suggest the following:
1) Make future controllers have as much in-hardware sane behavior as possible when unknown conditions are encountered. Error checking, basically. Users can input data on a web form, that’s why there’s error checking. Everyone knows users do stupid things. Clearly some of the people programming Intel EEPROMs for motherboard OEMs do stupid things too. What is sane default behavior? EEPROM error encountered = adapter shutdown and error message. Give the user notification and provide some mechanism for EEPROM validation and management...
2) Put more EEPROM validation in operating system drivers. Intel maintains ethernet drivers for various platforms. Why aren’t these drivers doing more validation of the adapter EEPROM? If my EEPROM was so badly misconfigured, why couldn’t the e1000e module have discovered that and notified me?
3) Produce and support an external tool for EEPROM testing, programming, and updating. In the course of working with Intel last fall I was provided a version of this tool for my testing so I know it exists. While I can understand why you don’t want random users messing with their EEPROM (and causing potential support nightmares) it seems the benefits would clearly outweigh any potential problems (of which there are already plenty).
The reality is Intel has no idea how many systems are affected by this issue or could be affected by issues like it. How could they? They’re expecting motherboard OEMs to follow their instructions (and understandably so). Just look at the combination of variables required to reproduce this issue:
- Intel 82574L
- Various specific misconfigured bytes in the EEPROM
- An insanely specific packet with the right value at just the right byte, received at a specific time
While most people weren’t able to reproduce this issue with their controller and EEPROM combinations it did kick off various discussions of periodic, random, sporadic failures across a wide range of ethernet adapters and general computing weirdness. A quick Google search returns a wide assortment of complaints with these adapters (and others like it) from a whole slew of users. EEPROM corruption. Random adapter resets. Packet drops. Various latency issues. PCI bus scan errors. ASPM problems. The list goes on and on.
Perhaps the “packet of death” for a slightly misconfigured Intel 82579 (for example) is my packet shifted 20 bytes in one direction or the other. Who knows? Please, please, please Intel - lets do everyone a favor and get these EEPROMs under control. End users update firmware all of the time - routers, set-top boxes, sometimes even their cars! Why can’t we have some utility to make sure our ethernet adapters aren’t just waiting to freak out when they receive the wrong packet?
I don’t believe in magic, swamp gas, sun spots, or any of the other “explanations” offered for some of the random strange behavior we often see with these complex devices (ethernet adapters or otherwise). That’s why I spent so long working on this issue to find a root cause (well that and screaming customers). I, like anyone else, encounter bugs and general weirdness in devices and software everyday in my life. Most of the time how do I respond to these bugs? I reboot, shrug my shoulders, say “that was weird”, and move on. Meanwhile I know, deep down, that there is a valid explanation for what just happened. Just like there was with my ethernet controllers.
Even with the explanation offered by Intel we could go much deeper. Why these bytes at that location? Why this packet? What’s up with the “inoculation” effect of some of the values? There are still many unanswered questions.
I’ve enjoyed reading many others report their tales of “extreme debugging” with the digital devices in their lives. It seems I’m not the only one that isn’t always satisfied with saying “that was weird” and moving on.
I've said it before and I'll say it again - I love the internet!
Wednesday, February 6, 2013
Packets of Death
When you're done reading this check out my update! Experiencing similar ethernet controller issues but don't know where to start?
UPDATE: See the packets and find out if you're affected here.
UPDATE 2: Yes, I've reproduced this issue regardless of OS, ASPM state/settings, or software firewall settings. Obviously if you have a layer 2/3 firewall in front of an affected interface you'll be ok.
Packets of death. I started calling them that because that’s exactly what they are.
Star2Star has a hardware OEM that has built the last two versions of our on-premise customer appliance. I’ll get more into this appliance and the magic it provides in another post. For now let’s focus on these killer packets.
About a year ago we released a refresh of this on-premise equipment. It started off simple enough, pretty much just standard Moore’s Law stuff. Bigger, better, faster, cheaper. The new hardware was 64-bit capable, had 8X as much RAM, could accommodate additional local storage, and had four Intel (my preferred ethernet controller vendor) gigabit ethernet ports. We had (and have) all kinds of ideas for these four ports. All in all it was pretty exciting.
This new hardware flew through performance and functionality testing. The speed was there and the reliability was there. Perfect. After this extensive testing we slowly rolled the hardware out to a few beta sites. Sure enough, problems started to appear.
All it takes is a quick Google search to see that the Intel 82574L ethernet controller has had at least a few problems. Including, but not necessarily limited to, EEPROM issues, ASPM bugs, MSI-X quirks, etc. We spent several months dealing with each and every one of these. We thought we were done.
We weren’t. It was only going to get worse.
I thought I had the perfect software image (and BIOS) developed and deployed. However, that’s not what the field was telling us. Units kept failing. Sometimes a reboot would bring the unit back, usually it wouldn’t. When the unit was shipped back, however, it would work when tested.
Wow. Things just got weird.
The weirdness continued and I finally got to the point where I had to roll my sleeves up. I was lucky enough to find a very patient and helpful reseller in the field to stay on the phone with me for three hours while I collected data. This customer location, for some reason or another, could predictably bring down the ethernet controller with voice traffic on their network.
Let me elaborate on that for a second. When I say “bring down” an ethernet controller I mean BRING DOWN an ethernet controller. The system and ethernet interfaces would appear fine and then after a random amount of traffic the interface would report a hardware error (lost communication with PHY) and lose link. Literally the link lights on the switch and interface would go out. It was dead.
Nothing but a power cycle would bring it back. Attempting to reload the kernel module or reboot the machine would result in a PCI scan error. The interface was dead until the machine was physically powered down and powered back on. In many cases, for our customers, this meant a truck roll.
While debugging with this very patient reseller I started stopping the packet captures as soon as the interface dropped. Eventually I caught on to a pattern: the last packet out of the interface was always a 100 Trying provisional response, and it was always a specific length. Not only that, I ended up tracing this (Asterisk) response to a specific phone manufacturer’s INVITE.
I got off the phone with the reseller, grabbed some guys and presented my evidence. Even though it was late in the afternoon on a Friday, everyone did their part to scramble and put together a test configuration with our new hardware and phones from this manufacturer.
We sat there, in a conference room, and dialed as fast as our fingers could. Eventually we found that we could duplicate the issue! Not on every call, and not on every device, but every once in a while we could crash the ethernet controller. However, every once in a while we couldn’t at all. After a power cycle we’d try again and hit it. Either way, as anyone who’s tried to diagnose a technical issue knows the first step is duplicating the problem. We were finally there.
Believe me, it took a long time to get here. I know how the OSI stack works. I know how software is segmented. I know that the contents of a SIP packet shouldn’t do anything to an ethernet adapter. It just doesn’t make any sense.
Between packet captures on our device and packet captures from the mirror port on the switch we were finally able to isolate the problem packet. Turns out it was the received INVITE, not the transmitted 100 Trying! The mirror port capture never saw the 100 Trying hit the wire.
Now we needed to look at this INVITE. Maybe the userspace daemon processing the INVITE was the problem? Maybe it was the transmitted 100 Trying? One of my colleagues suggested we shutdown the SIP software and see if the issue persisted. No SIP software running, no transmitted 100 Trying.
First we needed a better way to transmit the problem packet. We isolated the INVITE transmitted from the phone and used tcpreplay to play it back on command. Sure enough it worked. Now, for the first time in months, we could shut down these ports on command with a single packet. This was significant progress and it was time to go home, which really meant it was time to set this up in the lab at home!
Before I go any further I need to give another shout out to an excellent open source piece of software I found. Ostinato turns you into a packet ninja. There’s literally no limit to what you can do with it. Without Ostinato I could have never gotten beyond this point.
With my packet Swiss army knife in hand I started poking and prodding. What I found was shocking.
It all starts with a strange SIP/SDP quirk. Take a look at this SDP:
v=0
o=- 20047 20047 IN IP4 10.41.22.248
s=SDP data
c=IN IP4 10.41.22.248
t=0 0
m=audio 11786 RTP/AVP 18 0 18 9 9 101
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:0 PCMU/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:9 G722/8000
a=rtpmap:9 G722/8000
a=fmtp:101 0-15
a=rtpmap:101 telephone-event/8000
a=ptime:20
a=sendrecv
Wireshark picture:
Yes, I saw it right away too. The audio offer is duplicated and that’s a problem but again, what difference should that make to an Ethernet controller?!? Well, if nothing else it makes the ethernet frame larger...
But wait, there were plenty of successful ethernet frames in these packet captures. Some of them were smaller, some were larger. No problems with them. It was time to dig into the problem packet. After some more Ostinato-fu and plenty of power cycles I was able to isolate the problem pattern (with a problem frame).
Warning: we’re about to get into some hex.
The interface shutdown is triggered by a specific byte value at a specific offset. In this case the specific value was hex 32 at 0x47f. Hex 32 is an ASCII 2. Guess where the 2 was coming from?
a=ptime:20
All of our SDPs were identical (including ptime, obviously). All of the source and destination URIs were identical. The only difference was the Call-IDs, tags, and branches. Problem packets had just the right Call-ID, tags, and branches to cause the “2” in the ptime to line up with 0x47f.
BOOM! With the right Call-IDs, tags, and branches (or any random garbage) a “good packet” could turn into a “killer packet” as long as that ptime line ended up at the right address. Things just got weirder.
While generating packets I experimented with various hex values. As if this problem couldn’t get any weirder, it does. I found out that the behavior of the controller depended completely on the value of this specific address in the first received packet to match that address. It broke down to something like this:
Byte 0x47f = 31 HEX (1 ASCII) - No effect
Byte 0x47f = 32 HEX (2 ASCII) - Interface shutdown
Byte 0x47f = 33 HEX (3 ASCII) - Interface shutdown
Byte 0x47f = 34 HEX (4 ASCII) - Interface inoculation
Bad:
Good:
When I say “no effect” I mean it didn’t kill the interface but it didn’t inoculate the interface either (more on that later). When I say the interface shutdown, well, remember my description of this issue - the interface went down. Hard.
With even more testing I discovered this issue with every version of Linux I could find, FreeBSD, and even when the machine was powered up complaining about missing boot media! It’s in the hardware; the OS has nothing to do with it. Wow.
To make matters worse, using Ostinato I was able to craft various versions of this packet - an HTTP POST, ICMP echo-request, etc. Pretty much whatever I wanted. With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death - and kill client machines behind firewalls!
I know I’ve been pointing out how weird this whole issue is. The inoculation part is by far the strangest. It turns out that if the first packet received contains any value (that I can find) other than 1, 2, or 3 the interface becomes immune from any death packets (where the value is 2 or 3). Also, valid ptime attributes are defined inpowers (edit: multiples) of
10 - 10, 20, 30, 40. Depending on Call-ID, tag, branch, IP, URI, etc
(with this buggy SDP) these valid ptime attributes line up perfectly.
Really, what are the chances?!?
All of a sudden it’s become clear why this issue was so sporadic. I’m amazed I tracked it down at all. I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. At least I hope I don’t...
I was able to get in touch with two engineers at Intel and send them a demo unit to reproduce the issue. After working with them for a couple of weeks they determined there was an issue with the EEPROM on our 82574L controllers.
They were able to provide new EEPROM and a tool to write it out. Unfortunately we weren’t able to distribute this tool and it required unloading and reloading the e1000e kernel module, so it wouldn’t be preferred in our environment. Fortunately (with a little knowledge of the EEPROM layout) I was able to work up some bash scripting and ethtool magic to save the “fixed” eeprom values and write them out on affected systems. We now have a way to detect and fix these problematic units in the field. We’ve communicated with our vendor to make sure this fix is applied to units before they are shipped to us. What isn’t clear, however, is just how many other affected Intel ethernet controllers are out there.
I guess we’ll just have to see...
UPDATE: See the packets and find out if you're affected here.
UPDATE 2: Yes, I've reproduced this issue regardless of OS, ASPM state/settings, or software firewall settings. Obviously if you have a layer 2/3 firewall in front of an affected interface you'll be ok.
Packets of death. I started calling them that because that’s exactly what they are.
Star2Star has a hardware OEM that has built the last two versions of our on-premise customer appliance. I’ll get more into this appliance and the magic it provides in another post. For now let’s focus on these killer packets.
About a year ago we released a refresh of this on-premise equipment. It started off simple enough, pretty much just standard Moore’s Law stuff. Bigger, better, faster, cheaper. The new hardware was 64-bit capable, had 8X as much RAM, could accommodate additional local storage, and had four Intel (my preferred ethernet controller vendor) gigabit ethernet ports. We had (and have) all kinds of ideas for these four ports. All in all it was pretty exciting.
This new hardware flew through performance and functionality testing. The speed was there and the reliability was there. Perfect. After this extensive testing we slowly rolled the hardware out to a few beta sites. Sure enough, problems started to appear.
All it takes is a quick Google search to see that the Intel 82574L ethernet controller has had at least a few problems. Including, but not necessarily limited to, EEPROM issues, ASPM bugs, MSI-X quirks, etc. We spent several months dealing with each and every one of these. We thought we were done.
We weren’t. It was only going to get worse.
I thought I had the perfect software image (and BIOS) developed and deployed. However, that’s not what the field was telling us. Units kept failing. Sometimes a reboot would bring the unit back, usually it wouldn’t. When the unit was shipped back, however, it would work when tested.
Wow. Things just got weird.
The weirdness continued and I finally got to the point where I had to roll my sleeves up. I was lucky enough to find a very patient and helpful reseller in the field to stay on the phone with me for three hours while I collected data. This customer location, for some reason or another, could predictably bring down the ethernet controller with voice traffic on their network.
Let me elaborate on that for a second. When I say “bring down” an ethernet controller I mean BRING DOWN an ethernet controller. The system and ethernet interfaces would appear fine and then after a random amount of traffic the interface would report a hardware error (lost communication with PHY) and lose link. Literally the link lights on the switch and interface would go out. It was dead.
Nothing but a power cycle would bring it back. Attempting to reload the kernel module or reboot the machine would result in a PCI scan error. The interface was dead until the machine was physically powered down and powered back on. In many cases, for our customers, this meant a truck roll.
While debugging with this very patient reseller I started stopping the packet captures as soon as the interface dropped. Eventually I caught on to a pattern: the last packet out of the interface was always a 100 Trying provisional response, and it was always a specific length. Not only that, I ended up tracing this (Asterisk) response to a specific phone manufacturer’s INVITE.
I got off the phone with the reseller, grabbed some guys and presented my evidence. Even though it was late in the afternoon on a Friday, everyone did their part to scramble and put together a test configuration with our new hardware and phones from this manufacturer.
We sat there, in a conference room, and dialed as fast as our fingers could. Eventually we found that we could duplicate the issue! Not on every call, and not on every device, but every once in a while we could crash the ethernet controller. However, every once in a while we couldn’t at all. After a power cycle we’d try again and hit it. Either way, as anyone who’s tried to diagnose a technical issue knows the first step is duplicating the problem. We were finally there.
Believe me, it took a long time to get here. I know how the OSI stack works. I know how software is segmented. I know that the contents of a SIP packet shouldn’t do anything to an ethernet adapter. It just doesn’t make any sense.
Between packet captures on our device and packet captures from the mirror port on the switch we were finally able to isolate the problem packet. Turns out it was the received INVITE, not the transmitted 100 Trying! The mirror port capture never saw the 100 Trying hit the wire.
Now we needed to look at this INVITE. Maybe the userspace daemon processing the INVITE was the problem? Maybe it was the transmitted 100 Trying? One of my colleagues suggested we shutdown the SIP software and see if the issue persisted. No SIP software running, no transmitted 100 Trying.
First we needed a better way to transmit the problem packet. We isolated the INVITE transmitted from the phone and used tcpreplay to play it back on command. Sure enough it worked. Now, for the first time in months, we could shut down these ports on command with a single packet. This was significant progress and it was time to go home, which really meant it was time to set this up in the lab at home!
Before I go any further I need to give another shout out to an excellent open source piece of software I found. Ostinato turns you into a packet ninja. There’s literally no limit to what you can do with it. Without Ostinato I could have never gotten beyond this point.
With my packet Swiss army knife in hand I started poking and prodding. What I found was shocking.
It all starts with a strange SIP/SDP quirk. Take a look at this SDP:
v=0
o=- 20047 20047 IN IP4 10.41.22.248
s=SDP data
c=IN IP4 10.41.22.248
t=0 0
m=audio 11786 RTP/AVP 18 0 18 9 9 101
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:0 PCMU/8000
a=rtpmap:18 G729/8000
a=fmtp:18 annexb=no
a=rtpmap:9 G722/8000
a=rtpmap:9 G722/8000
a=fmtp:101 0-15
a=rtpmap:101 telephone-event/8000
a=ptime:20
a=sendrecv
Wireshark picture:
Yes, I saw it right away too. The audio offer is duplicated and that’s a problem but again, what difference should that make to an Ethernet controller?!? Well, if nothing else it makes the ethernet frame larger...
But wait, there were plenty of successful ethernet frames in these packet captures. Some of them were smaller, some were larger. No problems with them. It was time to dig into the problem packet. After some more Ostinato-fu and plenty of power cycles I was able to isolate the problem pattern (with a problem frame).
Warning: we’re about to get into some hex.
The interface shutdown is triggered by a specific byte value at a specific offset. In this case the specific value was hex 32 at 0x47f. Hex 32 is an ASCII 2. Guess where the 2 was coming from?
a=ptime:20
All of our SDPs were identical (including ptime, obviously). All of the source and destination URIs were identical. The only difference was the Call-IDs, tags, and branches. Problem packets had just the right Call-ID, tags, and branches to cause the “2” in the ptime to line up with 0x47f.
BOOM! With the right Call-IDs, tags, and branches (or any random garbage) a “good packet” could turn into a “killer packet” as long as that ptime line ended up at the right address. Things just got weirder.
While generating packets I experimented with various hex values. As if this problem couldn’t get any weirder, it does. I found out that the behavior of the controller depended completely on the value of this specific address in the first received packet to match that address. It broke down to something like this:
Byte 0x47f = 31 HEX (1 ASCII) - No effect
Byte 0x47f = 32 HEX (2 ASCII) - Interface shutdown
Byte 0x47f = 33 HEX (3 ASCII) - Interface shutdown
Byte 0x47f = 34 HEX (4 ASCII) - Interface inoculation
Bad:
Good:
When I say “no effect” I mean it didn’t kill the interface but it didn’t inoculate the interface either (more on that later). When I say the interface shutdown, well, remember my description of this issue - the interface went down. Hard.
With even more testing I discovered this issue with every version of Linux I could find, FreeBSD, and even when the machine was powered up complaining about missing boot media! It’s in the hardware; the OS has nothing to do with it. Wow.
To make matters worse, using Ostinato I was able to craft various versions of this packet - an HTTP POST, ICMP echo-request, etc. Pretty much whatever I wanted. With a modified HTTP server configured to generate the data at byte value (based on headers, host, etc) you could easily configure an HTTP 200 response to contain the packet of death - and kill client machines behind firewalls!
I know I’ve been pointing out how weird this whole issue is. The inoculation part is by far the strangest. It turns out that if the first packet received contains any value (that I can find) other than 1, 2, or 3 the interface becomes immune from any death packets (where the value is 2 or 3). Also, valid ptime attributes are defined in
All of a sudden it’s become clear why this issue was so sporadic. I’m amazed I tracked it down at all. I’ve been working with networks for over 15 years and I’ve never seen anything like this. I doubt I’ll ever see anything like it again. At least I hope I don’t...
I was able to get in touch with two engineers at Intel and send them a demo unit to reproduce the issue. After working with them for a couple of weeks they determined there was an issue with the EEPROM on our 82574L controllers.
They were able to provide new EEPROM and a tool to write it out. Unfortunately we weren’t able to distribute this tool and it required unloading and reloading the e1000e kernel module, so it wouldn’t be preferred in our environment. Fortunately (with a little knowledge of the EEPROM layout) I was able to work up some bash scripting and ethtool magic to save the “fixed” eeprom values and write them out on affected systems. We now have a way to detect and fix these problematic units in the field. We’ve communicated with our vendor to make sure this fix is applied to units before they are shipped to us. What isn’t clear, however, is just how many other affected Intel ethernet controllers are out there.
I guess we’ll just have to see...
Friday, September 21, 2012
FreeSWITCH Stable Branch - Sponsored by Star2Star
As many of you know I’ve been quite fond of FreeSWITCH for some time now. I’ve been impressed with the functionality, stability, and performance. Did I say impressed? I suppose I meant to say thrilled. Some of my more long-term readers may remember this post from three years ago. I can’t believe it’s been that long. I suppose that makes me (officially) old.
Getting back on track (as I often have to do) I have had one concern with FreeSWITCH over the years - the lack of a stable branch. Traditionally (in software development) once a project or piece of code reaches a certain level of maturity (or use, even) there comes a need to segment the code into an independently maintained entity. In most revision control systems this is typically called a “branch”. Makes sense because lines of code are often referred to as “trees”. I get it!
Since the beginning of the project in 2006 all of FreeSWITCH development has take place in one central code repository (trunk - trees again). At any given point in time a user could pull down trunk and receive the latest and “greatest” code - new features, bug fixes, security patches, etc. Unfortunately these new features can introduce new bugs. They can re-expose old bugs. They can change behavior in unexpected ways.
One of the things that has been most impressive about the FreeSWITCH project, actually, has been the relative stability of this often new and untested code. It’s been extremely rare to find a serious issue in trunk. However, the mere knowledge of this code management practice (or relative lack thereof) and potential for new features/bugs is an issue.
I learned a long time ago that people expect their phones to just work. The same user who finds it perfectly acceptable to reboot their computer after a software crash will jump and scream when a telephone doesn’t work. I guess that’s a testament to the level of experience people have historically expected. Interestingly this expectation seems to be changing in the face of new technology, which is something I have covered before.
Anyway, for the time being people (including myself) expect these things to “just work”. Of course this expectation places a tremendous burden on network and facility operators such as myself (and rightfully so). Meeting this expectation with FreeSWITCH trunk has required a significant amount of testing, testing, re-testing, regression testing, patching, etc. We simply cannot unleash a new version of software in the field without significant testing. When issues are found we have to address them. This takes time and resources.
Meanwhile FreeSWITCH is an open source project. That can’t be expected to provide for every need and whim of large commercial users such as myself. Why should they? They already spend so much of their limited time and resources to provide software that essentially powers my business and (indirectly) provides for my livelihood - for free. With these limited resources alone they can’t be expected to make time for the maintenance of yet another collection of code.
That’s why I’m happy to announce the Star2Star sponsorship of the FreeSWITCH stable branch! For the past several months Star2Star has been providing the financial assistance necessary for the FreeSWITCH project to hire another full-time team member to not only maintain a stable branch (1.2 as of this writing) but improve documentation, packaging, and community interaction. Everyone at Star2Star (including myself) couldn’t be happier to provide this resource for the project and the community. We look forward to working more with the FreeSWITCH team on the stable branch and any other projects that may advance FreeSWITCH and the state of the art in communications!
Getting back on track (as I often have to do) I have had one concern with FreeSWITCH over the years - the lack of a stable branch. Traditionally (in software development) once a project or piece of code reaches a certain level of maturity (or use, even) there comes a need to segment the code into an independently maintained entity. In most revision control systems this is typically called a “branch”. Makes sense because lines of code are often referred to as “trees”. I get it!
Since the beginning of the project in 2006 all of FreeSWITCH development has take place in one central code repository (trunk - trees again). At any given point in time a user could pull down trunk and receive the latest and “greatest” code - new features, bug fixes, security patches, etc. Unfortunately these new features can introduce new bugs. They can re-expose old bugs. They can change behavior in unexpected ways.
One of the things that has been most impressive about the FreeSWITCH project, actually, has been the relative stability of this often new and untested code. It’s been extremely rare to find a serious issue in trunk. However, the mere knowledge of this code management practice (or relative lack thereof) and potential for new features/bugs is an issue.
I learned a long time ago that people expect their phones to just work. The same user who finds it perfectly acceptable to reboot their computer after a software crash will jump and scream when a telephone doesn’t work. I guess that’s a testament to the level of experience people have historically expected. Interestingly this expectation seems to be changing in the face of new technology, which is something I have covered before.
Anyway, for the time being people (including myself) expect these things to “just work”. Of course this expectation places a tremendous burden on network and facility operators such as myself (and rightfully so). Meeting this expectation with FreeSWITCH trunk has required a significant amount of testing, testing, re-testing, regression testing, patching, etc. We simply cannot unleash a new version of software in the field without significant testing. When issues are found we have to address them. This takes time and resources.
Meanwhile FreeSWITCH is an open source project. That can’t be expected to provide for every need and whim of large commercial users such as myself. Why should they? They already spend so much of their limited time and resources to provide software that essentially powers my business and (indirectly) provides for my livelihood - for free. With these limited resources alone they can’t be expected to make time for the maintenance of yet another collection of code.
That’s why I’m happy to announce the Star2Star sponsorship of the FreeSWITCH stable branch! For the past several months Star2Star has been providing the financial assistance necessary for the FreeSWITCH project to hire another full-time team member to not only maintain a stable branch (1.2 as of this writing) but improve documentation, packaging, and community interaction. Everyone at Star2Star (including myself) couldn’t be happier to provide this resource for the project and the community. We look forward to working more with the FreeSWITCH team on the stable branch and any other projects that may advance FreeSWITCH and the state of the art in communications!
Thursday, July 26, 2012
Performance Testing (Part 2) - Going Commercial
Some time ago I wrote the first installment of a multi-part series on performance testing. Here we go with part two!
In part one I talked about some of the difficulties surrounding performance testing - functionality, flexibility, high scale, quality metrics, etc.
After looking at a couple commercial products we discovered a few things:
1) Some of these products really do “have it all”.
2) They can be very expensive.
There was some initial sticker shock ($100,000+) but looking back I shouldn’t have been so surprised. My first reaction (of course) was to reach out to a few people in the open source community with a proposal. In a classic “build vs. buy” scenario I wanted to build. This is (roughly) what I needed:
- At least 40,000 RTP streams (20,000 calls)
- At least 100,000 SIP calls and/or registrations
- The ability to emulate multiple user agents (VLAN, MAC address, DHCP client, SIP UA)
- RTP stream analysis on each RTP leg (MOS, R-Factor, PESQ, etc)
- Flexible device emulation - SIP headers, supported SIP features, etc
- Multiple codec support (at least G711, G729, G722, SILK/OPUS with FEC et
all, etc).
- Control of test scenarios - CPS, number of calls, duration of call,
total duration of test, etc
- Ability to save/load tests via web interface (for ease of use,
comparison of results, etc)
- Ability to perform feature testing - generate DTMF during a call to
navigate an IVR, for example.
- A modular system for monitoring the device under test - Linux load,
CPU usage, disk usage, network I/O, etc. Could also monitor Cisco
switches in between devices, Windows hosts, etc. Maybe even
FreeSWITCH or Asterisk if that's what was running on the device under
test.
- Saving and graphing of all relevant performance data - call setup
time, delay, duration, RTP jitter, packet loss, RTP stream stats, etc.
Ability to save data and generate reports from said data.
- Scalable design with master/slave architecture to scale across hosts
with the ability for hardware.
Did I mention this tool needs to be usable by various test engineers, some of which don’t know the difference between SIP and SDP (and rightfully so, they shouldn’t need to)?
With the open source software already available I figured this could be made available for less than the cost of a commercial testing solution.
I gave it away with the title of this post but you can guess what happened next: it was going to cost far more to develop everything I needed. By the way - it would also take six months to build and take 10 years off my life hunting bugs, managing the project, etc.
BUY
For less than what it would cost to build everything above I could buy this. A multi-user chassis with one Xcellon-Ultra NP load module and room for another one. 180,000 emulated SIP endpoints. 96,000 RTP streams. Wire speed 10 gigabit VoIP testing (and 12 gigabit ports).
Of course this isn’t a perfect world... The chassis runs Windows. The client software is only available for Windows and the interface is probably the furthest from what I want. As a guy that eats, lives, and breathes CLI (and has for a decade) multi-pane/dropdown/hidden/shadow GUIs are NOT my thing. I don’t even know what to call or how to describe some of the window/GUI elements present in the IxLoad user interface...
Ixia even has a solution to this problem. They offer TCL scripting with clients for various platforms, including Linux! While we’ll eventually get into that for the time being we went with a much simpler solution: we setup a Windows terminal server. I use CoRD from my Mac, login to the terminal server, and run a test. As you’ll see in part three - IT JUST WORKS.
In part one I talked about some of the difficulties surrounding performance testing - functionality, flexibility, high scale, quality metrics, etc.
After looking at a couple commercial products we discovered a few things:
1) Some of these products really do “have it all”.
2) They can be very expensive.
There was some initial sticker shock ($100,000+) but looking back I shouldn’t have been so surprised. My first reaction (of course) was to reach out to a few people in the open source community with a proposal. In a classic “build vs. buy” scenario I wanted to build. This is (roughly) what I needed:
- At least 40,000 RTP streams (20,000 calls)
- At least 100,000 SIP calls and/or registrations
- The ability to emulate multiple user agents (VLAN, MAC address, DHCP client, SIP UA)
- RTP stream analysis on each RTP leg (MOS, R-Factor, PESQ, etc)
- Flexible device emulation - SIP headers, supported SIP features, etc
- Multiple codec support (at least G711, G729, G722, SILK/OPUS with FEC et
all, etc).
- Control of test scenarios - CPS, number of calls, duration of call,
total duration of test, etc
- Ability to save/load tests via web interface (for ease of use,
comparison of results, etc)
- Ability to perform feature testing - generate DTMF during a call to
navigate an IVR, for example.
- A modular system for monitoring the device under test - Linux load,
CPU usage, disk usage, network I/O, etc. Could also monitor Cisco
switches in between devices, Windows hosts, etc. Maybe even
FreeSWITCH or Asterisk if that's what was running on the device under
test.
- Saving and graphing of all relevant performance data - call setup
time, delay, duration, RTP jitter, packet loss, RTP stream stats, etc.
Ability to save data and generate reports from said data.
- Scalable design with master/slave architecture to scale across hosts
with the ability for hardware.
Did I mention this tool needs to be usable by various test engineers, some of which don’t know the difference between SIP and SDP (and rightfully so, they shouldn’t need to)?
With the open source software already available I figured this could be made available for less than the cost of a commercial testing solution.
I gave it away with the title of this post but you can guess what happened next: it was going to cost far more to develop everything I needed. By the way - it would also take six months to build and take 10 years off my life hunting bugs, managing the project, etc.
BUY
For less than what it would cost to build everything above I could buy this. A multi-user chassis with one Xcellon-Ultra NP load module and room for another one. 180,000 emulated SIP endpoints. 96,000 RTP streams. Wire speed 10 gigabit VoIP testing (and 12 gigabit ports).
Of course this isn’t a perfect world... The chassis runs Windows. The client software is only available for Windows and the interface is probably the furthest from what I want. As a guy that eats, lives, and breathes CLI (and has for a decade) multi-pane/dropdown/hidden/shadow GUIs are NOT my thing. I don’t even know what to call or how to describe some of the window/GUI elements present in the IxLoad user interface...
Ixia even has a solution to this problem. They offer TCL scripting with clients for various platforms, including Linux! While we’ll eventually get into that for the time being we went with a much simpler solution: we setup a Windows terminal server. I use CoRD from my Mac, login to the terminal server, and run a test. As you’ll see in part three - IT JUST WORKS.
Thursday, July 12, 2012
My Linux Story
While digging through some boxes the other day I came across a book. Not just any book - a very important book. A book I both cursed and loved more than any other book in my entire life.
To tell the story of the book I need to give a little background. My father was a professor at the University of Illinois at Chicago. When I was old enough (13) I would spend my summers volunteering in his department (Occupational Therapy) doing random IT jobs - making ethernet cables, cleaning up viruses, fixing printers, etc.
One day they took me on a tour of the other departments in his building. They had some medical visualization apps running in one of the departments. Here we were, sitting in a room, with some PhD students working on some of the coolest computers I’d ever seen. They had bright colors and huge monitors. They ran tons of advanced looking applications. Most of the interaction was through a very interesting command line interface. I was intrigued.
Being a rambunctious 13 year old I approached one of the students and complimented him on his computer. I also asked him how I could get one of them for my house. He laughed. At the time I didn’t understand why it was a funny question. Looking back now I know it’s because those “computers” were actually high powered Silicon Graphics Indigo workstations that cost upwards of $40,000 each (in 1997 dollars).
His response was a muffled “Ummm, you can’t”. Not being dissuaded my follow-up question was “How can I get something like it?”. One of the students looked at the other student. They both kind of shrugged until one of them said “umm, Linux?”. Still not being quite sure they both agreed that “Linux” would be my best bet. I asked them how I could get “Linux”. They told me to go down to the university book store and buy a book about Linux. A good Linux book, they said, should include a CD-ROM with a copy of Linux I could install on my computer at home.
That day, before we went to Union Station to catch the Metra, my dad and I went to the university bookstore to find a book about Linux. Sure enough we found a Linux book that included not one but TWO CD-ROMs! I read as much of the book as I could before I got home that night.
Once I was at home I booted the family computer (with a DOS bootdisk) and ran a loadlin (wow, remember that?) based install program from the CD-ROM (no el torito!). During the course of the install I repartitioned and reformatted the (only) family computer - an IBM Aptiva M51. It had a Pentium 100, 1GB hard drive, and 16MB of RAM (upgraded from 8MB). It also came with one of those HORRIBLE MWAVE adapters and some shady Cirrus Logic graphics (IIRC).
Anyway, the install process (which was pretty horrible, actually) left me with what was (at the time) a barely usable computer. How was my sister going to do her homework? How were we going to get on to the internet? Uh-oh, looks like I better learn more about this “Linux” thing...
So that’s how it started. Later that year (for Christmas) my parents realized they weren’t going to get the Aptiva back from me so we bought another computer to run Windows. At that point my little Linux workstation became my computer and the “gateway” for my first home network - a 10BASE2 (coax!) Ethernet network using NE2000 cards from Linksys. Internet access went through my Aptiva using an external modem that, regardless of type, could only negotiate a maximum of 19,200 bps on our crappy phone line. PPP, chat scripts, dial-on-demand, squid for caching, etc. 15 years later I’m still (primarily) using Linux!
After finding “My First Linux Book” I wondered what would happen if I tried to install that version of Linux today. Some people reminisce about their childhood by hearing certain songs, playing a sport, or collecting action figures. I (apparently) do it by installing ancient versions of Linux.
I needed to install this version of Linux but the CD-ROMs in my book were missing. I looked around the internet for a while but could not find an ISO or copy of the distro anywhere. I could barely find references to the book. Where else could I look? Everywhere else I look - Amazon!
Sure enough, Amazon had a used copy of the book (with CD-ROM) for $5 with shipping. Two days later it was here. To my surprise the book (and the CDs) were in excellent condition. Who the hell is keeping a warehouse full of mid-nineties Linux books (3rd edition with Kernel 2.0!)?
I get on my main development machine at home, download a FreeDOS ISO (to install from, remember), and create a VirtualBox virtual machine. What should it look like? I decide to “go big”:
Keep in mind I’m going to be running kernel 2.0 here - this hardware needs to be supported by a 15 year old kernel. I get Unifix Linux 2.0 installed. Moments later I’m logged into my “new” Linux system. Not knowing exactly what to do now, I decide to try to get networking to work.
Long story short I could not get Linux 2.0 to recognize the emulated Am79C973 ethernet controller. I tried changing the device ids and recompiling the kernel (takes less than one minute, btw) but couldn’t get it to work.
Hmmm, what else could I do for connectivity? Maybe I could go really nostalgic and get something running over the serial port?
I configured VirtualBox to emulate a 16550 serial port as COM1. I setup VirtualBox to point the other end of the emulated serial port to a local pipe. I figured that if I could somehow run pppd on both sides of this serial port (host and guest) and configure NAT I could get this thing on the internet.
Here’s how I did it:
1) Launch socat to convert the unix domain socket provided by VirtualBox to a standard Linux tty so pppd can run on it:
(on host)
socat UNIX-CONNECT:[com1] PTY,link=[vmodem0],raw,echo=0,waitslave
Where [com1] is the path to your VirtualBox socket and [vmodem0] is the path to your (new) tty.
2) Launch pppd on the new tty:
(on host)
pppd [vmodem0] 57600 192.168.100.1:192.168.100.2 nodetach local
Once again where [vmodem0] is the path to your new socat tty. Make sure that the IP addresses provided for each end of the PPP link don’t conflict with any local IP addresses.
3) Setup kernel iptables on the host:
(on host)
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i ppp0 -o eth0 -j ACCEPT
4) Connect the virtual machine to the host:
(on guest)
pppd /dev/ttyS0 57600 defaultroute passive
Sure enough, here’s what I saw on the host:
Using interface ppp0
Connect: ppp0 <--> /home/kris/projects/linux_universe/vmodem0
local IP address 192.168.100.1
remote IP address 192.168.100.2
Boom! 1990s Linux, meet the 21st century!
Once I had networking up and running things really took off. I noticed all of the services running by default on my old Linux host (portmap, yp, apache, telnet, echo, chargen, sendmail, wu-ftpd, etc). Remember the 90s when the internet wasn’t such a hostile place!?!
Here’s some fun command output from my “new” host old_linux:
root@old_linux:~ # uname -a
Linux old_linux 2.0.25 Unifix-2.0 i686
root@old_linux:~ # ping -c 5 192.168.100.1
PING 192.168.100.1 (192.168.100.1): 56 data bytes
64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=15.1 ms
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=19.9 ms
--- 192.168.100.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 15.1/18.9/19.9 ms
root@old_linux:~ #
20ms of latency on the same physical machine (pppd -> VirtualBox -> socat -> pppd)!
root@old_linux:~ # ping -c 5 www.google.com
PING www.l.google.com (74.125.139.104): 56 data bytes
64 bytes from 74.125.139.104: icmp_seq=0 ttl=45 time=59.5 ms
64 bytes from 74.125.139.104: icmp_seq=1 ttl=45 time=49.9 ms
64 bytes from 74.125.139.104: icmp_seq=2 ttl=45 time=49.9 ms
64 bytes from 74.125.139.104: icmp_seq=3 ttl=45 time=50.5 ms
64 bytes from 74.125.139.104: icmp_seq=4 ttl=45 time=69.9 ms
--- www.l.google.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 49.9/55.9/69.9 ms
root@old_linux:~ # gcc -v
Reading specs from /usr/lib/gcc-lib/i486-unknown-linux/2.7.2.1/specs
gcc version 2.7.2.1
root@old_linux:~ # httpd -v
Server version Apache/1.1.1.
root@old_linux:~ # ssh -v
SSH Version 1.2.14 [i486-unknown-linux], protocol version 1.4.
Standard version. Does not use RSAREF.
Pre-iptables. Pre-ipchains. IPFWADM!
root@old_linux:~ # ipfwadm -h
ipfwadm 2.3.0, 1996/07/30
root@old_linux:/usr/src/linux-2.0.25 # time make
[trimmed output]
The new kernel is in file arch/i386/boot/bzImage; now install it as vmlinuz
A name list of the kernel is in vmlinux.map; you may install it as vmlinux
9.86user 11.74system 0:22.78elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+0minor)pagefaults 0swaps
root@old_linux:/usr/src/linux-2.0.25 #
23 seconds to compile the kernel!
Ok, but what about the modules?
make[1]: Leaving directory `/usr/src/linux-2.0.25/arch/i386/math-emu'
8.80user 7.42system 0:16.80elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+0minor)pagefaults 0swaps
root@old_linux:/usr/src/linux-2.0.25 #
16 seconds to compile the modules!
I have to admit I’m fascinated with this distro and the state of Linux as I was first introduced to it. Of course some of the memories have faded over time but more than anything it’s amazing to see how far we’ve come...
1997:
2012:
...and this machine is pretty old at this point! Needless to say with 256MB of RAM (double the maximum possible in my Aptiva) and even one emulated CPU Unifix 2.0 barely knows what to do with this new hardware (even if it isn’t real)!
While I was never quite sure why I was doing any of this I can tell you that it was very fun to remember how all of this got started. I remember my next distro being RedHat 5.2...
To tell the story of the book I need to give a little background. My father was a professor at the University of Illinois at Chicago. When I was old enough (13) I would spend my summers volunteering in his department (Occupational Therapy) doing random IT jobs - making ethernet cables, cleaning up viruses, fixing printers, etc.
One day they took me on a tour of the other departments in his building. They had some medical visualization apps running in one of the departments. Here we were, sitting in a room, with some PhD students working on some of the coolest computers I’d ever seen. They had bright colors and huge monitors. They ran tons of advanced looking applications. Most of the interaction was through a very interesting command line interface. I was intrigued.
Being a rambunctious 13 year old I approached one of the students and complimented him on his computer. I also asked him how I could get one of them for my house. He laughed. At the time I didn’t understand why it was a funny question. Looking back now I know it’s because those “computers” were actually high powered Silicon Graphics Indigo workstations that cost upwards of $40,000 each (in 1997 dollars).
His response was a muffled “Ummm, you can’t”. Not being dissuaded my follow-up question was “How can I get something like it?”. One of the students looked at the other student. They both kind of shrugged until one of them said “umm, Linux?”. Still not being quite sure they both agreed that “Linux” would be my best bet. I asked them how I could get “Linux”. They told me to go down to the university book store and buy a book about Linux. A good Linux book, they said, should include a CD-ROM with a copy of Linux I could install on my computer at home.
That day, before we went to Union Station to catch the Metra, my dad and I went to the university bookstore to find a book about Linux. Sure enough we found a Linux book that included not one but TWO CD-ROMs! I read as much of the book as I could before I got home that night.
Once I was at home I booted the family computer (with a DOS bootdisk) and ran a loadlin (wow, remember that?) based install program from the CD-ROM (no el torito!). During the course of the install I repartitioned and reformatted the (only) family computer - an IBM Aptiva M51. It had a Pentium 100, 1GB hard drive, and 16MB of RAM (upgraded from 8MB). It also came with one of those HORRIBLE MWAVE adapters and some shady Cirrus Logic graphics (IIRC).
Anyway, the install process (which was pretty horrible, actually) left me with what was (at the time) a barely usable computer. How was my sister going to do her homework? How were we going to get on to the internet? Uh-oh, looks like I better learn more about this “Linux” thing...
So that’s how it started. Later that year (for Christmas) my parents realized they weren’t going to get the Aptiva back from me so we bought another computer to run Windows. At that point my little Linux workstation became my computer and the “gateway” for my first home network - a 10BASE2 (coax!) Ethernet network using NE2000 cards from Linksys. Internet access went through my Aptiva using an external modem that, regardless of type, could only negotiate a maximum of 19,200 bps on our crappy phone line. PPP, chat scripts, dial-on-demand, squid for caching, etc. 15 years later I’m still (primarily) using Linux!
After finding “My First Linux Book” I wondered what would happen if I tried to install that version of Linux today. Some people reminisce about their childhood by hearing certain songs, playing a sport, or collecting action figures. I (apparently) do it by installing ancient versions of Linux.
I needed to install this version of Linux but the CD-ROMs in my book were missing. I looked around the internet for a while but could not find an ISO or copy of the distro anywhere. I could barely find references to the book. Where else could I look? Everywhere else I look - Amazon!
Sure enough, Amazon had a used copy of the book (with CD-ROM) for $5 with shipping. Two days later it was here. To my surprise the book (and the CDs) were in excellent condition. Who the hell is keeping a warehouse full of mid-nineties Linux books (3rd edition with Kernel 2.0!)?
I get on my main development machine at home, download a FreeDOS ISO (to install from, remember), and create a VirtualBox virtual machine. What should it look like? I decide to “go big”:
- 1 CPU
- 256MB of RAM
- 8GB Intel PIIX3 hard drive
- 1 Am79C973 ethernet port
Keep in mind I’m going to be running kernel 2.0 here - this hardware needs to be supported by a 15 year old kernel. I get Unifix Linux 2.0 installed. Moments later I’m logged into my “new” Linux system. Not knowing exactly what to do now, I decide to try to get networking to work.
Long story short I could not get Linux 2.0 to recognize the emulated Am79C973 ethernet controller. I tried changing the device ids and recompiling the kernel (takes less than one minute, btw) but couldn’t get it to work.
Hmmm, what else could I do for connectivity? Maybe I could go really nostalgic and get something running over the serial port?
I configured VirtualBox to emulate a 16550 serial port as COM1. I setup VirtualBox to point the other end of the emulated serial port to a local pipe. I figured that if I could somehow run pppd on both sides of this serial port (host and guest) and configure NAT I could get this thing on the internet.
Here’s how I did it:
1) Launch socat to convert the unix domain socket provided by VirtualBox to a standard Linux tty so pppd can run on it:
(on host)
socat UNIX-CONNECT:[com1] PTY,link=[vmodem0],raw,echo=0,waitslave
Where [com1] is the path to your VirtualBox socket and [vmodem0] is the path to your (new) tty.
2) Launch pppd on the new tty:
(on host)
pppd [vmodem0] 57600 192.168.100.1:192.168.100.2 nodetach local
Once again where [vmodem0] is the path to your new socat tty. Make sure that the IP addresses provided for each end of the PPP link don’t conflict with any local IP addresses.
3) Setup kernel iptables on the host:
(on host)
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A FORWARD -i eth0 -o ppp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i ppp0 -o eth0 -j ACCEPT
4) Connect the virtual machine to the host:
(on guest)
pppd /dev/ttyS0 57600 defaultroute passive
Sure enough, here’s what I saw on the host:
Using interface ppp0
Connect: ppp0 <--> /home/kris/projects/linux_universe/vmodem0
local IP address 192.168.100.1
remote IP address 192.168.100.2
Boom! 1990s Linux, meet the 21st century!
Once I had networking up and running things really took off. I noticed all of the services running by default on my old Linux host (portmap, yp, apache, telnet, echo, chargen, sendmail, wu-ftpd, etc). Remember the 90s when the internet wasn’t such a hostile place!?!
Here’s some fun command output from my “new” host old_linux:
root@old_linux:~ # uname -a
Linux old_linux 2.0.25 Unifix-2.0 i686
root@old_linux:~ # ping -c 5 192.168.100.1
PING 192.168.100.1 (192.168.100.1): 56 data bytes
64 bytes from 192.168.100.1: icmp_seq=0 ttl=64 time=15.1 ms
64 bytes from 192.168.100.1: icmp_seq=1 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=2 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=3 ttl=64 time=19.9 ms
64 bytes from 192.168.100.1: icmp_seq=4 ttl=64 time=19.9 ms
--- 192.168.100.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 15.1/18.9/19.9 ms
root@old_linux:~ #
20ms of latency on the same physical machine (pppd -> VirtualBox -> socat -> pppd)!
root@old_linux:~ # ping -c 5 www.google.com
PING www.l.google.com (74.125.139.104): 56 data bytes
64 bytes from 74.125.139.104: icmp_seq=0 ttl=45 time=59.5 ms
64 bytes from 74.125.139.104: icmp_seq=1 ttl=45 time=49.9 ms
64 bytes from 74.125.139.104: icmp_seq=2 ttl=45 time=49.9 ms
64 bytes from 74.125.139.104: icmp_seq=3 ttl=45 time=50.5 ms
64 bytes from 74.125.139.104: icmp_seq=4 ttl=45 time=69.9 ms
--- www.l.google.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 49.9/55.9/69.9 ms
root@old_linux:~ # gcc -v
Reading specs from /usr/lib/gcc-lib/i486-unknown-linux/2.7.2.1/specs
gcc version 2.7.2.1
root@old_linux:~ # httpd -v
Server version Apache/1.1.1.
root@old_linux:~ # ssh -v
SSH Version 1.2.14 [i486-unknown-linux], protocol version 1.4.
Standard version. Does not use RSAREF.
Pre-iptables. Pre-ipchains. IPFWADM!
root@old_linux:~ # ipfwadm -h
ipfwadm 2.3.0, 1996/07/30
root@old_linux:/usr/src/linux-2.0.25 # time make
[trimmed output]
The new kernel is in file arch/i386/boot/bzImage; now install it as vmlinuz
A name list of the kernel is in vmlinux.map; you may install it as vmlinux
9.86user 11.74system 0:22.78elapsed 94%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+0minor)pagefaults 0swaps
root@old_linux:/usr/src/linux-2.0.25 #
23 seconds to compile the kernel!
Ok, but what about the modules?
make[1]: Leaving directory `/usr/src/linux-2.0.25/arch/i386/math-emu'
8.80user 7.42system 0:16.80elapsed 96%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+0minor)pagefaults 0swaps
root@old_linux:/usr/src/linux-2.0.25 #
16 seconds to compile the modules!
I have to admit I’m fascinated with this distro and the state of Linux as I was first introduced to it. Of course some of the memories have faded over time but more than anything it’s amazing to see how far we’ve come...
1997:
- Linux 2.0.25
- Original Pentium at 100MHz
- 16MB RAM
- 1GB hard drive
- 1MB VRAM
2012:
- Linux 3.2.0 (Ubuntu 12.04)
- Quad-core Pentium Xeon at 2.27GHz
- 12GB RAM
- 1TB hard drive x2 (RAID 1)
- 1GB VRAM (dual 1080p displays)
...and this machine is pretty old at this point! Needless to say with 256MB of RAM (double the maximum possible in my Aptiva) and even one emulated CPU Unifix 2.0 barely knows what to do with this new hardware (even if it isn’t real)!
While I was never quite sure why I was doing any of this I can tell you that it was very fun to remember how all of this got started. I remember my next distro being RedHat 5.2...
Subscribe to:
Posts (Atom)