Update on the $15 Cavium SNIC10E 10GbE card: prebuilt image, second-host benchmarks, and I need beta testers
retained text
A month ago I posted about turning the Cavium CN6640-SNIC10E into a working dual-port 10GbE card with no vendor firmware. Here's what changed since, plus a clarification and a request. WHAT'S NEW \- Prebuilt card image in the releases. No more building OpenWrt for hours: download the image, run install.sh. \- The host module now installs through DKMS, so it survives kernel upgrades. \- Card temperatures show up in sensors with no serial cable, plus an estimated power draw. \- Rewritten quick start, one block from git clone to a working oct0. Hardcoded paths are gone. \- Requirements fixed: "Above 4G decoding" is NOT needed. Secure Boot has to be off. SECOND-HOST RESULTS Same card re-measured on a different machine and peer (Dell R720 + HP NC523): 9.64 Gb/s TX and 9.93 Gb/s RX aggregate across both ports, matching the first rig. Each port alone reaches line rate, but the two together share about 10 Gb/s per direction, because the card sits on a PCIe Gen2 x4 link. A CLARIFICATION ON "NOTHING FLASHED" The card OS always runs from RAM and is never written to flash. There are two ways to boot it: \- Over serial (boot-clean.sh): nothing is written to the card at all. Power it off and it's exactly as it was. \- Serial-free (the default): a one-time step saves the u-boot environment to NAND, so the host can boot the card on its own. That is a write, but restore-bootapp.sh puts the stock env back. So "fully reversible" holds either way, and "nothing flashed" is strictly true only for the serial path. The docs now say this plainly. KNOWN ISSUES \- Heavy TX collapses RX under full duplex. Still hunting the cause. \- ARP broadcasts sometimes don't leave the card after a reload (static neighbour entries work around it). \- Hot-plugging a DAC can leave TX stuck until the service is restarted. BETA TESTERS WANTED So far this has only run on my own card. If you have an SNIC10E in a drawer, I'd love to know if it works on yours. You need a Linux host with Secure Boot off and a USB-serial adapter for the one-time provisioning. Useful feedback: host model, kernel version, whether install.sh worked first try, and iperf3 numbers. Report here, even just to say "it worked": https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic/discussions/1 Repo: https://github.com/nicologiuliani6/cavium-cn6640-snic10e-octeon-ii-nic Docs: https://nicologiuliani.site/docs/cavium-cn6640-snic10e/ Write-up: https://nicologiuliani.site/blog/cavium-cn6640-bar2-datapath/ SMALL SPOILER After the first post someone reached out asking if the same approach could open up a Sophos XGS 107w firewall. I've been working on it and it's almost done. More soon. If you want to support the work, GitHub Sponsors is here: https://github.com/sponsors/nicologiuliani6
document fetched 2026-10-04 00:32 · canonical https://reddit.com/r/homelab/comments/1wwsioc/update_on_the_15_cavium_snic10e_10gbe_card