Abstract: Accurately predicting 3D structures and dynamics of physical systems is crucial in scientific applications. Existing approaches that rely on geometric Graph Neural Networks (GNNs) effectively enforce $\mathrm{E}(3)$-equivariance, but they often fail in leveraging extensive broader information. While direct application of Large Language Models (LLMs) can incorporate external knowledge, they lack the capability for spatial reasoning with guaranteed equivariance. In this paper, we propose EquiLLM, a novel framework for representing 3D physical systems that seamlessly integrates $\mathrm{E}(3)$-equivariance with LLM capabilities. Specifically, EquiLLM comprises four key components: geometry-aware prompting, an equivariant encoder, an LLM, and an equivariant adapter. Essentially, the LLM guided by the instructive prompt serves as a sophisticated invariant feature processor, while 3D directional information is exclusively handled by the equivariant encoder and adapter modules. Experimental results demonstrate that EquiLLM delivers significant improvements over previous methods across molecular dynamics simulation, human motion simulation, and antibody design, highlighting its promising generalizability.

Lay Summary:

Predicting how things move and interact in 3D, like molecules or people, is really important for many scientific fields. Current AI methods either excel at understanding 3D geometry but miss important background knowledge, or they have vast knowledge but struggle with spatial reasoning.We developed EquiLLM, a new AI framework that combines the best of both worlds. Our approach uses large language models to tap into extensive scientific knowledge, while specialized components handle the complex 3D geometry calculations. The framework works by having the language model process scientific information and context, while dedicated modules ensure all spatial calculations remain mathematically consistent.EquiLLM significantly improves capabilities in areas like simulating how molecules move, understanding human motion, and even designing antibodies. This shows that EquiLLM is a versatile tool that can be applied to many different scientific challenges.