In millimeter wave (mmWave) communications, aligning the transmitter and receiver beams is crucial to reduce the significant path loss. As scanning the entire directional space is inefficient, designing an efficient and robust method to identify the correct optimal beamforming direction is essential. Many existing works use bandit algorithms for beam alignment but rely on unimodality or multimodality assumptions on the reward structure, as well as assuming the horizon is sufficiently long. However, such assumptions may not hold in practice and cause such algorithms to converge to choosing suboptimal beams.In this work, we propose the physics-informed algorithms GLM-ETC and GLM-Greedy that exploit the existence of a dominant path (e.g., LoS path), an assumption that is perhaps more realistic in practice.Through simulated experiments using the DeepMIMO dataset (Alkhateeb, 2019), we demonstrate that both algorithms outperform existing approaches across 4,952 bandit instances.