[P] Can we remove BatchNorm and LayerNorm? Our new method retains 76% Top-1 on CIFAR-100 at batch size 1.
retained text
Hi everyone, I'm sharing our new paper: **"Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers."** We explore a simple question: *Can learned geometry replace feature normalization?* The core idea is **SALU** (Saturated Adaptive Linear Unit), which replaces normalization layers (BatchNorm, LayerNorm) with a learnable element-wise transformation using just **2 learnable scalars per layer**, without computing activation statistics (no mean/variance). To put it simply: * Normalization moves the distribution. * Derf moves the function. * **SALU deforms the function around a fixed center.** **Key Results:** * **CIFAR-100 (ResNet-18):** Achieves **83.42% Top-1** at BS=512, and **retains 76.23% at BS=1** (while standard BN collapses). * **CIFAR-100 (ViT):** Achieves **70.80%** Top-1 * **ImageNet-1K (from scratch):** * ResNet-50: **80.16%** Top-1 * ViT-B/16: **80.28%** Top-1 (using only BS=128!) * **Hardware:** Trained on a single shared NVIDIA A40 (24 GB). **Links:** * 📄 Paper (arXiv): [https://arxiv.org/abs/2606.02927](https://arxiv.org/abs/2606.02927) * 💻 Code, configs & pretrained checkpoints: [https://github.com/MouradZaied/SaluNet](https://github.com/MouradZaied/SaluNet) * 🧵 Summary thread on X: [https://x.com/Mourad\_ZAIED/status/2119487366918480035](https://x.com/Mourad_ZAIED/status/2119487366918480035) I am very open to collaboration, especially on: * Matched baselines and additional seeds. * Large-batch ImageNet training with SALU + SWALU. * Small-batch ResNet-18 repetitions. Happy to answer any questions in the comments!
document fetched 2026-11-10 00:23 · canonical https://reddit.com/r/deeplearning/comments/1xrqi5f/p_can_we_remove_batchnorm_and_layernorm_our_new