Poster
in
Workshop: Methods and Opportunities at Small Scale (MOSS)
Foundation Models on a Budget: Approximating Blocks in Large Vision Models
Irene Cannistraci · Simone Antonelli · Emanuele Palumbo · Thomas Sutter · Emanuele Rodola · Bastian Rieck · Julia Vogt
Keywords: [ classification ] [ neural network similarities ] [ foundation models ] [ latent representations ] [ large models ] [ representation learning ]
Foundation Models have shown impressive performance in various tasks and domains, yet they require massive computational resources, raising concerns about accessibility and sustainability. In this paper, we propose Transformer Blocks Approximation (TBA), a novel method that leverages intra-network similarities to identify and approximate transformer blocks in large vision models using only a small amount of training data. TBA replaces these blocks using lightweight, closed-form transformations, without any additional training steps. The proposed method reduces the number of parameters while having minimal impact on the downstream task.