← sources

6 years of selfhosting, hit with RAM corruption

source: reddit (reddit:selfhosted)published 2026-11-13 13:43ingested 2026-11-14 02:38

retained text

My Proxmox host kernel panicked and hung. I assumed it was a Intel GVT driver bug at first, I used LLMs to debug this issue, cause i've never faced something like this before. I want people to learn from my mistakes.

Journal had nothing. The panic record was in pstore (`/var/lib/systemd/pstore`): a general protection fault on pointer `0xffbf8…` that should have been `0xffff8…`. That's one flipped bit, bit 54. Digging back, I found 9 October oopses on a different kernel build with the same flipped bit. A BIOS update didn't help; it panicked again a day later, same bit.

The machine is a Lenovo tiny PC with non-ECC SO-DIMMs, so there were only crashes.

# Files affected

In my Docker VM, a Firefox-based container segfaulted on every launch. `dpkg --verify` flagged a library. Read with `dd iflag=direct`, which bypasses the page cache, the file matched the package. The cached copy had 8 bytes flipped. Same bit.

Two files were also corrupted on disk: a kernel module, and a Perl library with 61 bytes changed, every one at bit 54. Anything written while the RAM was bad, package upgrades included, can carry flips.

# ZFS didn't save me

ZFS, PBS digests and rsync checksums all protect data *after* it's hashed. Flip a bit before hashing and the corruption gets checksummed and stored forever. One backup failed with "detected chunk with wrong digest" because memory changed between hashing and upload. The scary ones are the backups that didn't fail: months of them verify perfectly and could still be wrong.

# Verification sweep

I moved the disks to other hardware and checked everything, read-only:

* `dpkg --verify` on the host, every LXC, VM and every container
* Postgres dumps, MariaDB `CHECK EXTENDED`, `PRAGMA integrity_check` on 29 SQLite DBs, Paperless checksums
* ISOs against vendor SHA-256
* `find -newerct` to list what landed on the NAS during the bad window. Photos: nothing.

All clean except those two files, now reinstalled.

# Backups

My rsync job to a USB drive had been failing for a month. TrueNAS raised an alert; drive was also 98% full and i've been delaying dealing with it. Turns out that saved me.

I attached it read-only and hashed 26k files against the NAS. All identical.

Bonus: rsync without `--delete` had kept my old PBS datastore *including pruned snapshots*, from before the RAM went bad. PBS chunks verify themselves: the SHA-256 of the decompressed data is the filename. So a small Python script checked 45,449 chunks (78 GB) offline. All good. Those are now my known-good restore points.

# Lesson learnt: Avoid Rsync, use Restic/Borg for future. Make sure backups work, have more than one copy, ECC does matter.

Almost certainly bad RAM (memtest still pending). There was little damage on disk. But the backups on PBS need to be completely checked.

# Lessons

1. Set `kernel.panic=10` so it reboots instead of hanging
2. Make sure you have > 1 Backup and Test it
3. Check pstore after a crash
4. Same bit flipped in every oops? Suspect hardware
5. Checksums can't fix corruption upstream. Get ECC if you can
6. Keep old snapshots before pruning after an incident
7. Read your NAS alerts

With these RAM prices, I retired the machine, moved my PBS to a VM on TrueNAS and made the PBS machine my PVE host.

document fetched 2026-11-14 00:29 · canonical https://reddit.com/r/selfhosted/comments/1xv883c/6_years_of_selfhosting_hit_with_ram_corruption