I mean that to connect to a service you wouldn't need to know the port, the IPv6 address would be enough.

This is why I consider ports a layer violation of sorts. You never talk to a machine with TCP/UDP, you talk to a service on a machine. And so as it is the full address to the service isn't just the layer 3 address.

As I mentioned this would be especially interesting when hosting multiple services, same or different, on the same machine since there would be no port conflict.