HOW TWINAX CABLES RUINED MY DAY

Thursday January 19, 2017  

The day actually started out pretty well. The weather was nice, I’d had my morning coffees, and I was expecting some new firewalls to arrive. I was especially excited about this point.

You see, I had spent the last few weeks working on a new network design. I had the hardware picked out. The topology was looking good. I even had my cable maps drawn up. Everything was going well.

I’m sitting at my desk half-heartedly working on a document. When would the ASA’s arrive? Would I have to wait another day? I’m trying to convince myself that it’s not that big a deal when I hear the beep-beep-beep of a truck reversing outside my window. Could this be it? I jumped up and looked out the window. Garbage truck… Well, I guess we need them too.

Hours go by as the clock moves from 9:30 to 10 am. Then I hear another truck. I look out the window… definitely a courier this time… And several large boxes with Ciscoprinted on the side are unloaded. I’m trying to play it cool. After all, I should really be excited about a few new firewalls turning up should I? I’m a grown man after all. Shouldn’t there be more exciting things in my life? I’ll work that one out another time. Right now, I have a few new ASA’s to unbox!

Seven boxes. Four ASA 5555’s and three additional rail kits. I’m still not sure why there are four ASA’s and three additional rail kits. Somehow they ended up on the purchase order. Did the supplier slip them in under the radar? Wait. Stay on target. That’s a problem for another time.

A colleague of mine, whom I shall call Tim Jim, is there to help with the unboxing. Together, we get the ASA’s out and install them in some pre-provisioning racks. They were racked right above some shiny new Nexus switches, which we had configured a few days earlier. Jim slides in behind the rack to begin cabling.

The lighting is poor behind the pre-prod racks, and it’s a tight squeeze to get back there. It can be funny to watch people trying to navigate the narrow dark claustrophobic corridor. Trying not to trip over the cables the last person has left hanging. Hoping that the power cables are not live.

I find it even funnier that Jim’s about the size of the Statue of Liberty. And that all the equipment is racked about 3RU from the floor. Not only does he have to get behind the rack, he has to crawl.

Anyway, I’m getting off topic. The ASA’s have an additional interface card pre-installed. It comes with six GE SFP ports. We wanted this interface card, as the Nexus has all SFP+ ports, no RJ45. We used Twinax cables for data and RJ45 for management. There’s a 2248 FEX for all the management ports.

It’s all cabled up. Console ports connected to a breakout box. We’re ready to go. I log onto the switches and the ASA’s. Then the power cuts out. It’s only a fraction of a second, but it’s enough for the switches to reboot. As it turns out, one of the UPS’s in the pre-prod room has bad batteries. It filters power surges, but can’t keep anything turned on for more than 0.1 seconds.

The timing’s bad, but that’s ok. Look on the bright side; I’ve got my new toys to play with.

OK, let’s bring a few interfaces up. Line protocol down, huh? Oh, that’s right. The Nexus has 10G ports, and the ASA has 1G. Let’s set the speed to 1000 on the switch ports…

ERROR: Speed is 1G, but the transceiver doesn’t support this speed

Hold on, what? Sure, the Nexus has 10G ports, but I should be able to set them to 1G. Right? Right?

…

Oh, no… I’ve messed up…

Chest tightens. Breathing becomes laboured. Ego deflates.

See where I went wrong? The Twinax cables may be electrical, but they’re really just SFP’s. Well, they’re 10G, so they’re technically SFP+.

Do you see it now? SFP’s run a particular speed. A 1G SFP runs at 1Gig. A 10G SFP runs at 10Gig. A 40G SFP… Well, you get the idea. What this means is that when an SFP is inserted into a switch port, the switch port will only run at the speed that the SFP supports. That means that the 10G Twinax cable simply can’t be slowed down to 1G.

I don’t have time for this! We’re on a very tight schedule. The customer needs their network upgraded. This is going to set me back. No, no, no, no, this is going to set the whole team back. The server guys can’t even start until I’m done. I need to breathe into a paper bag.

While I’m feeling ice daggers in my heart, Jim keeps a cool head. He’s good at this mostly because he has no hair for insulation. He’s also quite good at crisis management, and before long, we’re digging up 1G Multimode SX SFPs from the spare parts bin. Maybe it’s not sp bad after all? Just replace a few cables and SFP’s, and we’re ok…

Hang on. We’re not done yet. When connecting the ASA to the Nexus, the Nexus port would come up, but the ASA’s would not. It would not budge from Line Protocol Down. On top of that, every time the switch port is manually shut/no shut, the ASA would pop up an error:

Reached an autonegotiation error limit of 10 for B/D/F=20/0/1

I’m riding the rollercoaster of emotion. Time to call TAC. A nice guy from the Sydney TAC was able to help me out. He suggested running the no negotiate auto command on the Nexus switch port. What do you know? it works! We now have a way to move forward.

It’s such as simple thing. Twinax is a cable with SFP’s built in. It can’t have its speed changed. I’ve been networking for years, and somehow I tripped over something so simple.

It’s been 30 hours since this happened, and I’m finally starting to breathe again.

Had a similar experience? Post a comment below.