Modern vision assistants often rely on large, static backbones regardless of input complexity, leading to unnecessary energy use and latency—especially on edge devices. We introduce Adaptive Backbone Selection (ABS), a dynamic inference framework that selects the most appropriate CNN backbone for each image in real-time. ABS integrates a lightweight complexity analyzer (based on edge and texture richness) and a policy network, trained via reinforcement learning, that learns to dynamically balance accuracy and latency through a custom reward function. To mitigate switching overhead, a memory-efficient Backbone Manager with LRU caching handles model reuse. Evaluated on ImageNet, ABS establishes a new, superior operating point on the accuracy-efficiency frontier, achieving higher accuracy than strong baselines like DenseNet121 at a fraction of the computational cost. Our work presents a practical and deployable system for building more sustainable and responsive real-time AI.