Earlier this year, Mike Joseph from Meta gave a very interesting talk at NANOG 94 in Denver. The title of his talk was “PON in the Datacenter: Hyperscale for Management and Console” (slides here). Here’s a 41-minute video of this talk, and it is very much worth watching if you are in the datacenter / ISP space.
Notes from Meta’s talk:
XGS PON seems good enough to replace copper for management traffic. Again, this is for management traffic only and not the actual Meta content traffic, which likely would be over 10G/40G/100G links within the datacenter.
By using PON (Passive Optical Network), Meta has designed to terminate copper coming from devices within the rack so that outside of the rack is just (lighter) optical fibre (only). Copper, just for management alone at their scale, was quite a lot of cables. An end-of-row device mounted a “goalpost rack” on top. They had a rigid design to plan for Ethernet & serial drops in advance for each rack. Changes in that were harder due to the fixed design of copper cables.
With PON, they were able to get (20+20 - ethernet & console drops) or any other combination like (32+8) per rack. On a row basis, that’s up to 1280 Ethernet drops + 800 console connections - where a pair of them in a rack would aggregate within the rack.
They didn’t say the vendor name, but from the pictures, it’s quite clear they are using Cienna’s solution. The picture below shows 6 x 4 port ONUs stacked, giving 24 copper ports. This can be expanded to 12 x 4 ports = 48 ports easily, as visible on the chassis.
Meta also has a high availability on the PON network by making use of 2 x OLTs feeding a 2:8 or 2:4 splitter.
They also have an interesting headend. Instead of using traditional OLT, they are using “micro-OLTs”. These are OLTs implemented on an SFP+ module and plug into a layer 3 switch. I think from the picture (plus the fact they are using Cienna ONUs), this is Cienna’s Tibit pluggables.
Image source: Cienna Tibit.
These devices basically speak 10G Ethernet at one end and XGS PON on the other end. In the backend, they do Ethernet to PON MAC bridge.
Thoughts on PON outside of the last mile
I found this presentation fascinating because I have always wondered why PON is not being deployed outside of the FTTH last mile. I have asked this to a few friends working on enterprise networks and felt as if the product line is not developed yet, while technically it’s very much feasible, as we know from the FTTH world.
Go to any corporate office with 200-250 people sitting on each floor. These typically deploy Cat6 copper from each desk to a centralised switch. That’s a lot of copper, and the aggregation of that takes quite a bit of expensive real estate. A 4-port OLT with a 64-user per port can easily do 4 x 64 = 256 users. Likely much higher than that because optical power (unlike FTTH networks) won’t be an issue here. A mix of active Ethernet (for high traffic) + PON + DC power (to power ONUs) running over the same physical cable + pre-terminated fibre can do quite a bit. Corning displayed these concepts at the Courtyard hotel 6 years ago (video here).
Once, I was at a hotel in Singapore, and at night, it was crazy loud right outside of my room. One can easily tell it was a switch placed in a telco closet. All these can be quite nice applications for PON going forward. Would save a ton of expensive, heavy copper Cat6 cable, along with real estate, electricity and more when offering drops. An XGS 10G symmetric uplink/downlink capacity can easily do 150Mbps per drop for a 1:64 split or 78Mbps for a 1:128 split. Most of the hotel’s Wi-Fi is inbound heavy (streaming effect) while cameras are outbound heavy. Even at 10Mbps per stream, it can do many hundreds of cameras on the same PON port, where copper aggregates within a few meters on fibre and fibre aggregates per floor on the splitters.
I think going forward, we will see wifi APs as well as IP cameras, printers, and IP phones with PON ONU (or even PON based ATAs like this one!) tech built in, and it will essentially become a plug-and-play solution with pre-terminated hybrid copper-fibre, where copper will carry DC power and fibre will carry traffic.
