← sources

[P] Can we remove BatchNorm and LayerNorm? Our new method retains 76% Top-1 on CIFAR-100 at batch size 1.

source: reddit (reddit:deeplearning)published 2026-11-09 12:33ingested 2026-11-10 02:38

retained text

Hi everyone,

I'm sharing our new paper: **"Geometry, Not Statistics: SaluNet for Normalization-Free CNNs and Transformers."**

We explore a simple question: *Can learned geometry replace feature normalization?*

The core idea is **SALU** (Saturated Adaptive Linear Unit), which replaces normalization layers (BatchNorm, LayerNorm) with a learnable element-wise transformation using just **2 learnable scalars per layer**, without computing activation statistics (no mean/variance).

To put it simply:

* Normalization moves the distribution.
* Derf moves the function.
* **SALU deforms the function around a fixed center.**

**Key Results:**

* **CIFAR-100 (ResNet-18):** Achieves **83.42% Top-1** at BS=512, and **retains 76.23% at BS=1** (while standard BN collapses).
* **CIFAR-100 (ViT):** Achieves **70.80%** Top-1
* **ImageNet-1K (from scratch):**
   * ResNet-50: **80.16%** Top-1
   * ViT-B/16: **80.28%** Top-1 (using only BS=128!)
* **Hardware:** Trained on a single shared NVIDIA A40 (24 GB).

**Links:**

* 📄 Paper (arXiv): [https://arxiv.org/abs/2606.02927](https://arxiv.org/abs/2606.02927)
* 💻 Code, configs & pretrained checkpoints: [https://github.com/MouradZaied/SaluNet](https://github.com/MouradZaied/SaluNet)
* 🧵 Summary thread on X: [https://x.com/Mourad\_ZAIED/status/2119487366918480035](https://x.com/Mourad_ZAIED/status/2119487366918480035)

I am very open to collaboration, especially on:

* Matched baselines and additional seeds.
* Large-batch ImageNet training with SALU + SWALU.
* Small-batch ResNet-18 repetitions.

Happy to answer any questions in the comments!

document fetched 2026-11-10 00:23 · canonical https://reddit.com/r/deeplearning/comments/1xrqi5f/p_can_we_remove_batchnorm_and_layernorm_our_new