ETH Zurich's Optmax Moves AI Attention Math From Silicon to Light
A research team proposes using thin-film lithium niobate modulators as analog replacements for Softmax and Sigmoid inside transformer attention.
- Researchers propose Optmax and Optmoid, analog replacements for Softmax and Sigmoid using lithium niobate modulators.
- Softmax is under 1% of ops but disproportionately bottlenecks transformer inference latency on digital hardware.
- The modulator's sinusoidal voltage-to-light curve approximates the exponential and normalization in Softmax.
- ViT on MNIST, CIFAR-10, SVHN and GPT-2 on FineWeb-Edu match digital baselines after drop-in replacement.
- Accuracy survives aggressive 4-bit input-output quantization and realistic multiplicative noise at 10 GBaud.
- Targeted as a hybrid co-packaged nonlinear accelerator alongside existing digital tensor cores, not a full optical chip.
Researchers from ETH Zurich, Queen’s University, and collaborating institutions propose moving Softmax and Sigmoid-style attention activations from GPU arithmetic to analog photonic units. Their paper introduces Optmax and Optmoid, two functions implemented with thin-film lithium niobate Mach-Zehnder modulators. In the reported Vision Transformer and GPT-2 experiments, models trained with these approximations remained close to digital baselines.
The study combines measurements from fabricated modulators with hardware-aware model simulations. An integrated GPU-photonic accelerator and measured end-to-end speedup remain future work, so the latency and energy results are projections rather than system benchmarks.
Why Softmax outruns its FLOP count
Softmax converts each row of attention scores into normalized weights using exponentials and a rowwise sum:
softmax(xi) = exp(xi) / Σj exp(xj)
GPU tensor cores process dense matrix multiplications at high throughput, while exponentials rely on a smaller pool of special-function hardware. Softmax also requires reductions, synchronization, and division. Those dependencies can put the operation on the critical path even when it represents less than 1% of a transformer’s arithmetic.
Standard full attention scales quadratically with sequence length because each token interacts with every other token. Fused implementations such as tiled attention kernels reduce memory traffic and avoid materializing the full attention matrix, but they still compute row maxima, exponentials, sums, and normalized outputs.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
