Poster
in
Workshop: ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models
Graph Signal Processing Meets Mamba2: Adaptive Filter Bank via Delta Modulation
Yehjin Shin · Seojin Kim · Noseong Park
Abstract:
State-space models (SSMs) provide efficient alternatives to attention with linear-time recurrence. Mamba2, a recent SSM-based language model, employs selective input gating and a multi-head structure for parallel computation and strong performance. However, its multi-head recurrence operates independently without structured utilization or analysis. In this work, we introduce \textbf{H}ierarchical \textbf{AD}aptive filter bank for \textbf{E}fficient \textbf{S}SMs (\textit{HADES}), a GSP-inspired framework that reinterprets Mamba2 as an adaptive filter bank on a line graph. \textit{HADES} features two filter types: shared filters for global low-pass behavior and expert filters for local high-pass behavior, achieved via structured bias on parameter $\Delta$. \textit{HADES} matches baseline models, including Mamba2, on key benchmark tasks while using only \textbf{58.9\%} of the original parameters.
Chat is not available.