Protein surface fingerprint encodes chemical and geometric features that govern protein–protein interactions and can be used to predict changes in binding affinity between two protein complexes. Current state-of-the-art models for predicting binding affinity change, such as GearBind, are all-atom based geometric models derived from protein structures. Although surface properties can be implicitly learned from the protein structure, we hypothesize that explicit knowledge of protein surfaces can improve a structure based model's ability to predict changes in binding affinity. To this end, we introduce Pi-SAGE, a novel Permutation-Invariant Surface-Aware Graph Encoder. We first train Pi-SAGE to create a protein surface codebook directly from the structure and assign a token for each surface exposed residue. Next, we augmented the node features of the GearBind model with surface features from domain adapted Pi-SAGE to predict binding affinity change on the SKEMPI dataset. We show that explicitly incorporating local, context-aware chemical properties of residues enhances the predictive power of all-atom graph neural networks in modeling binding affinity changes between wild-type and mutant proteins.