Abstract: Recent advancements in large language models (LLMs) have significantly enhanced the ability of LLM-based systems to perform complex tasks through natural language processing and tool interaction. However, optimizing these LLM-based systems for specific tasks remains challenging, often requiring manual interventions like prompt engineering and hyperparameter tuning.Existing automatic optimization methods, such as textual feedback-based techniques (*e.g.*, TextGrad), tend to focus on immediate feedback, analogous to using immediate derivatives in traditional numerical gradient descent.However, relying solely on such feedback can be limited when the adjustments made in response to this feedback are either too small or fluctuate irregularly, potentially slowing down or even stalling the optimization process. In this paper, we introduce $\textbf{REVOLVE}$, an optimization method that tracks how $\textbf{R}$esponses $\textbf{EVOLVE}$across iterations in LLM systems. By focusing on the evolution of responses over time, REVOLVE enables more stable and effective optimization by making thoughtful, progressive adjustments at each step. Experiments across three tasks demonstrate the adaptability and efficiency of our proposal. Beyond its practical contributions, REVOLVE highlights a promising direction, where the rich knowledge from established optimization principles can be leveraged to enhance LLM systems, which paves the way for further advancements in this hybrid domain. Code is available at: https://llm-revolve.netlify.app.

Lay Summary:

Getting AI models (like chatbots or multi-agent assistants) to improve consistently can be challenging. Sometimes they get stuck or their progress is unstable, especially when dealing with complex tasks. Current methods often only look at the immediate success or failure of the latest attempt, missing valuable information about the learning process.Our paper presents REVOLVE, a new method for improving these AI models. Instead of just focusing on the last response, REVOLVE analyzes how the AI's answers change over multiple tries. By understanding this "evolution" or trend in the responses – similar to how engineers might consider momentum or the curve of progress in physical systems – REVOLVE makes smarter decisions about how to guide the AI's next steps.REVOLVE helps AI models learn more smoothly and reliably, avoiding common pitfalls. More importantly, it shows that powerful ideas from traditional optimization (the mathematical field of finding the best solutions) can be successfully adapted for the world of AI. This opens up exciting new research directions, suggesting that we can leverage decades of optimization knowledge to significantly boost the capabilities of future AI systems.