Thursday December 15, 2016
40G in the DC (Or, How I Learned to Stop Worrying and Love BiDi)
The Need for Speed
Have you ever felt dismay that your 10G network is reaching capacity? Or perhaps you’ve felt the dread of uncertainty that your design does not provide enough bandwidth?You are not alone.
For a while now, it’s been common to run 10G on host ports in the access layer (or leaf, depending on your topology). Higher load on the access layer means more throughput in the distribution layer. This is one reason why there’s an increasing trend to move to 40G in the data centre.
I felt this in a case where I needed to support an extra rack of servers, running a dense virtual machine load. This rack was not next to the rest of the equipment, so I decided to put a pair of 10G Nexus FEX at the top of the rack.
This is in a data centre, so each FEX uplink requires a cross connect, which incurs a monthly charge. Wanting to limit the number of cross connects, 40G seemed like the best solution. At least until I read a little more on how 40G fibre works.
10G vs 40G
10G and 40G are a bit different. 10G SR transcievers use a pair of multi-mode fibre (MMF) cores, and have LC connectors. It is likely that your 10G data centre network uses short MMF runs between racks.
Traditional 40G SR trancievers are a different story. 40G SR uses an MMF ribbon, containing 8 to 12 fibre cores, and use an MPO-12 connector. The difference in cabling requirements is what makes it so hard to move from 10G to 40G. The MMF ribbon uses four pairs of cores, each one 10G capable. As shown in the image below, in practice these ribbons contain 12 cores, four of which are wasted.
BiDi to the Rescue
If you want 40G, but don’t want to replace your fibre, you’re in luck. BiDi (Bi-Directional) optics can save your fibre runs. BiDi transceivers use 2 core multi-mode fibre with LC connectors. This makes it suitable for reusing your existing fibre plant.
BiDi uses two 20G bi-directional channels across two fibre cores. Each core has two 20G wavalengths (one for send and one for receive) at the same time. This results in 40G full duplex transmission across the fibre pair.
BiDi supports a run of up to 100 metres (328 feet) on OM3 multi-mode, or 150 metres (492 feet) on OM4.
Conclusion
BiDi offers a relatively cheap and easy solution to run 40G across racks that are close together. With a limit of 150 metres, it’s not made for runs between data halls.
This solution uses multi-mode fibre which is simple if you run your own fibre between racks. If you need data centre provided cross connects, make sure they support MMF. Some data centres only support SMF cross connects.
You may need 40G over a longer run, or perhaps MMF is not an option. In this case, an alternative is to use a non-BiDi 40G SFP for single-mode fibre. Have a look at the QSFP-40G-LR4 or QSFP-40G-LR4-S.