I think the more logical concern would be that it would wait until it could find any potential unsecured wifi to use. I live in an apartment building and there a few wifi access points that I can reach from my apartment with no security. Though one of them appears to be an air conditioner??
Say the TV finds such an access point and uses it. It phones home and so now the manufacturer now knows that such and such serial # TV is located in that neighborhood.
They still need to tie it to you. If you've not given it a shred of personal information, it has to be fingerprinting via surreptitious audio/video recording, or else by obtaining the purchase info from the retailer (like the name and billing address from your credit card) and matching that by serial number (which would have to be recorded during purchase).