SDL is a relatively high level API and framebuffer style graphics work fine on it. I think there are some edge cases for X11 with non-compositing window manager where you will get interesting glitches if the window moves.