True, but you can still do a confused deputy attack. The fraudster hires a property manager, informs them that they would like to remove the sign because they wish the list the property for sale. Either that or they con a realtor they're working with into doing it. The unknowing realtor, eager for the commission, knows a guy who can take it down.
There's always something that can happen in any scenario. Social engineering, hiring locals, deeper forms of identity theft, or worse. The possibilities never hit 0, they just become a lot less profitable (and a lot riskier) a scam to try to run.
For those like me who didn't know: https://en.wikipedia.org/wiki/Confused_deputy_problem
The idea is just to avoid being the softest target. The scammers attempting this fraud don't want to do all the work you describe. They'll just move on to the next vacant property.