loading
loading
AWS invited us to tour the networking lab where it develops its own switches, timing appliances, optical modules, and fiber connectors. We flew out to an unmarked building a few weeks ago, where Sam and I got out of the car wondering if we had the correct address. The tour covered switch generations from 3.2Tbps to 102.4Tbps, hardware encryption at line rate, a private time distribution network accurate to 20 microseconds, DWDM modules that put 64 wavelengths on one fiber strand, hollow-core fiber with roughly 33 percent better latency, connectors that install 64 fibers in about one second, and even a molded cover for ganged fibers that acts as a dust cover and loopback. Since Sam and I had to fly to the lab and AWS covered our travel expenses, we have to say this is sponsored. Also, since it was a lab tour, we were asked not to film many things, and there are others covered in the background. I asked AWS to review the footage to ensure we were not disclosing anything unintended. Since some items the reviewers did not know AWS planned to disclose, we will use only frames from the approved video footage for this release. Make no mistake, there is a lot of new stuff in the video that you will not have seen elsewhere, and if you are into networking, timing, and hyperscale, it is probably the most interesting video of 2026. Certainly share it with your colleagues. AWS Switching to 102.4Tbps The first bench holds the AWS in-house switch family, spanning several generations. Some of what we will show you are silver, unpainted lab chassis. I asked about the coloring, and apparently it seems to align with the Pantone color of the year. The blue chassis is a 3.2 Tbps switch with 32 QSFP28 ports at 100GbE each. It is already an older design at this point, and you can see that from inside. The green box next to it is the current volume deployment, a 12.8 Tbps switch where every port has moved from 100G QSFP28 to 400G QSFP-DD. This is also the generation that moved signaling from NRZ to PAM4. Opening the chassis makes the power increase obvious. The ASIC gets a larger heatsink, and the QSFP56 cages get their own larger heatsinks as well. Inside there is an Annapurna Labs management processor and an SSD for logging. A small board with a big job AWS pointed out one small internal board as a key part of reliability in the newer switches and optical modules. The board communicates with every pluggable optics module over I2C and collects telemetry, so operators can identify a failing module before it affects its neighbors. We were told this board may look like an ordinary PCB, but inside AWS it counts as a major innovation because of the telemetry information it provides and how it accomplishes that. AWS 51.2Tbps with 800G OSFP ports AWS is deploying the 51.2 Tbps generation heavily at the time of filming. The front panel carries OSFP ports running PAM4-112 SerDes, which gives 800GbE per port. For STH readers, Octal for eight, then 112G for roughly 100Gbps per lane. The heatsink grows again because two copies of the I2C telemetry board sit under it alongside the switch ASIC. The rear holds the management processor, power distribution and fan control. Chassis colors track generations in this lab. Blue covers the 3.2T, green covers the 12.8T, and we were told this one is officially peach. 102.4 Tbps with 1.6T ports The next step up runs at 102.4 Tbps and is not in volume deployment yet. AWS says it will roll out over the coming quarters and years. Each OSFP port uses PAM4-224 SerDes for a raw 1.6 Tbps. That is sixteen times the per-port bandwidth of the blue switch, on twice as many ports, so cooling grows accordingly. At these speeds, the optical modules include their own built-in cooling, which we have seen on other platforms at this performance level. Production units are expected in a whitish color, completing a palette of blue, green, peach and white. You can also see the management, I2C, and SSD logging in here, with the power delivery becoming more robust. Encryption at line rate The last switch on the bench handles a different problem. Every port on this 12.8 Tbps unit encrypts traffic, on the reasoning that data leaving a data center should be encrypted by default. The same chassis also serves as long-haul equipment because its ports accept high-power 400ZR optics, so coherent modules can plug straight in and drive fiber over metro distances. Sixteen FPGAs handle encryption at line rate, which explains the two-rack-unit chassis. More silicon means more power and more cooling. A neat one here is that the Annapurna Labs management processor is mounted flipped relative to the other designs on the bench. You can also see the I2C board and the logging SSD here. The management network AWS also builds its own 1GbE management switch. A 1GbE switch is not exotic, but AWS told us that the management plane is what makes a large network operable at their scale. The board carries PHYs, transformers, a switch chip, a management processor and a logging SSD. It is the same architecture as the high-speed boxes at far lower bandwidth. Alongside it sits an AWS-built serial console server. Yes, AWS even designed its own console server. Whatever else changes in the network, out-of-band access still runs on 1970s-era serial technology. A separate timing network The appliance on the next stand keeps time across fleets of hundreds of thousands of servers. Satellites provide reference clocks, and AWS distributes a pulse-per-second signal downstream over a dedicated timing network that is independent of its data and management networks. The front panel resembles a 48-port SFP switch, but these ports do not run Ethernet. The time distribution protocol is proprietary to AWS. Multiple antenna inputs provide redundancy. Four ports face upstream, and the rest face downstream so the network can be built as a tree that reaches every device. AWS then needs a map of the tree, plus the cable lengths and speeds of each link, to calibrate timing for each device. Accurate time matters for ordering. Distributed databases need write timestamps to guarantee that reads return current data, and log analysis is only as reliable as its clock sources. Standard NTP delivers roughly 500 microseconds of accuracy. AWS claims its timing infrastructure guarantees time to within 20 microseconds, and we were told that figure is conservative. Inside the appliance, two FPGAs monitor the pluggables over I2C, and a third runs the proprietary timing logic. An SSD, an Annapurna Labs processor, power distribution, and a fan control board fill out the rest. Next, let us get to optics, connectors, and a special part.
document fetched 2026-10-15 02:58 · canonical https://servethehome.com/inside-the-secretive-aws-networking-lab