I do usually worry - because DNS spoofing is still possible and we are one step (eg: a compromised certificate) away from being pwned. But yeah one shouldn't have to worry.