The adaptability of test-time adaptation is influenced by multiple real-world factors, including continual domain shifts and temporally correlated/imbalanced distributions.To address this, we propose a general SwiTTA framework for both CNNs and ViTs, featuring two key components: (1) a domain router with multiple domain experts performing online domain identification via feature statistics analysis, and (2) CFA - a temporal correlation handler employing contextual feature aggregation through sliding window averaging.Extensive experiments demonstrate that SwiTTA achieves state-of-the-art performance across diverse realistic scenarios,outperforming existing methods by significant margins.