Neither C nor C++ allows you to derive pointers pointing outside the original object (as in, memory allocation), it's UB. Yes, the arithmetic operation itself has UB, not merely later derefering an OoB pointer. The only exception is one-past-the-end pointers which are safe to create (but not to dereference of course).

Whether it's safe to "smuggle" pointers around as ints, without doing any arithmetic on them, is "complicated" and honestly there's no one single answer as far as I know.