My first TrueNAS Scale failure - oof
retained text
I've used TrueNAS for years, and never had an issue. I thought I knew all the basics to always be able to work things out. Oh, how wrong I was today. Precursor to today, last week we had a power cut, the UPS didn't hold out and it crashed out. The system is a header unit with 24 drives, and a secondary array of 36 in an expansion. With the power cut, the power came back on and the array just came up bad. The header came up fine, but marked the expansion of disks as bad. A reboot of the expansion allowed me to reset the drives to online and healthy, and then pull in the pool, all by CLI. Today, we changed the subnet of the network, which requires NIC changes. Something wouldn't stick, so I decided to just reboot the header... Which was the beginning of 3 hrs of misery.... It booted, but a battery error appeared stating that 7 disks were in the wrong mode (cached Vs write-through), and on the front of the header, 8 LEDS were out. I counted the disks in the POST boot sequence, they all showed, but every boot ended in a kernel panic while at a specific point that looked like a zpool mount. One weird thing though, the expansion array was going absolutely berserk like it was having a rave with the LEDS. I pondered. I consulted posts on here. I trawled issues. I asked el Claudius. Claude had an idea and one that I had already gotten to - why not boot without the expansion on and see what happens? Guess what, flawless boot. I powered back up the array, scanned the disks and boom, off went the disco. It turns out that something to do with a pool scrub kicking in during mount was causing it to fail during the boot sequence. I've never seen that one before. Scrub paused, everything mounted and added to the system (zpool imported) and then scrub unpaused. No idea why. But at least it's back. If anyone has any ideas, it's TrueNAS scale, and I'm all ears.
document fetched 2026-10-11 00:33 · canonical https://reddit.com/r/sysadmin/comments/1x2pv0p/my_first_truenas_scale_failure_oof