ArXiv

Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

2026-05-12 · 17:59 UTC ·Kexuan Shi, Hanxuan Li, Zeju Qiu... ·1 min read
Authors
Kexuan Shi, Hanxuan Li, Zeju Qiu...
Categories
cs.LG, stat.ML
arXiv
https://arxiv.org/abs/2605.12492v1
PDF
https://arxiv.org/pdf/2605.12492v1

Brief

Pion introduces an orthogonal equivalence transformation optimizer that modulates weight-matrix geometry by applying left/right orthogonal updates while strictly preserving singular values and the spectral norm. The paper derives the Pion update rule, studies design options and convergence properties, and presents empirical evidence that Pion is a stable, competitive alternative to additive optimizers (e.g., Adam, Muon) for LLM pretraining and fine-tuning.

Why it matters

Pion is a spectrum-preserving optimizer that updates each weight matrix via left and right orthogonal transformations, which preserve all singular values and therefore keep the matrix spectral norm fixed during training.

Key details

  • The authors (Kexuan Shi, Hanxuan Li, Zeju Qiu, Yandong Wen, Simon Buchholz, Weiyang Liu) derive Pion's update rule, analyze convergence and design choices, and report that Pion yields stable, competitive results on large language model pretraining and finetuning (technical report v1, 30 pages, 19 figures; arXiv:2605.12492v1, published 2026-05-12).
Source evidence

Abstract

We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

Comment: Technical report v1 (30 pages, 19 figures, project page: https://spherelab.ai/pion/)