I remember seeing a wildcat in a sanctuary in Costa Rica with a similar story (the kittens looked like housecat kittens, so someone adopted them, but the adult cat's behavior was different enough to be really challenging). I've been trying to remember the exact species I saw there (maybe an oncilla?), but apparently there are several kinds of wildcat fairly easily mistaken for housecats as kittens.

I think not all of them categorically refuse to be pets, but they're noticeably more aggressive and less affectionate compared to housecats.