Abstract: Modern last-mile delivery (LMD) systems are increasingly challenged by dynamic and unpredictable customer requests, which can disrupt route consistency, reduce operational efficiency, and increase mental load for couriers. To mitigate these issues, we introduce R$^3$S$^2$Route, a Robust Regularizer-enhanced RL-based framework designed to generate smooth and stable delivery routes under state uncertainty. Our approach leverages an actor-critic architecture and incorporates a robustness regularization term that penalizes excessive policy variation in response to input fluctuations. We define formal metrics for route smoothness and stability to quantify courier-friendliness, embedding these objectives directly into the learning process to promote spatially coherent and temporally consistent routing behavior. Experimental evaluations show that R$^3$S$^2$Route improves route smoothness by up to 59.68\% and route stability by 14.29\%, while preserving competitive performance in terms of travel distance and time window adherence, outperforming multiple baselines in dynamic LMD settings.