> But who decides which rules are neutral

You’re correct that rules are themselves to an high degree arbitrary. They can only be imposed by a source of authority that’s beyond question: God(s), elders, founding fathers, kings, etc. In the case of the U.S. it’s historical inertia going back to 1215.