There is a field of hierarchical RL in which the optimisation occurs over a range of time scales/abstraction. But I'm not aware of much practical success for these approaches so far.