Effectively navigating large, changing actionspaces is a critical challenge for reinforcement-learning orchestrators that rely on a non-stationarycatalogue of APIs, tool endpoints, LLMs, and aux-iliary agents. Each tool has a human-readable de-scription — e.g. its input schema, intended func-tionality, benchmark scores and output format -typically ignored by stanard RL. We introduceand analyse a family of semantic context RL al-gorithms that embed the semantic context of eachtool alongside the state of the environment andlearn policies that generalize between tool ver-sions, additions, and removals. We conclude withempirical studies of the role of semantic contextin linear bandits and LLM in-context learning.