Continual Test-Time Adaptation (CTTA) faces challenges when real-world domains are dynamic—recurring with varying frequencies and durations—unlike the structured changes many methods assume. Existing approaches then struggle with convergence issues from brief domain exposures, catastrophic forgetting, and knowledge misapplication in these dynamic conditions. We propose DPCore, a robust and computationally efficient method designed for such dynamic patterns. DPCore integrates three key components: Visual Prompt Adaptation for efficient domain alignment, a Prompt Coreset for knowledge preservation, and a Dynamic Update mechanism that intelligently manages prompts based on domain similarity. Extensive experiments on four benchmarks show DPCore achieves state-of-the-art performance in both structured and dynamic settings, significantly reducing trainable parameters by 99% and computation time by 64% compared to previous approaches.