Born with it or learned? Not everyone has it, not even beautiful women or gorgeous men. I've encountered plain people, men & women, who have it. I've also encountered very pretty people who don't have an ounce of it. I don't think looks play a role. I wouldn't have said that 20 yrs. ago.
Like you, I do not think it goes with physical appearance, despite what popular media tells us. I do not really find "model perfect" men appealing, not even when I was younger.
I think sex appeal goes way beyond the looks anyway. Myself, I am FAR from what people would normally find sexy or attractive. Neither is my husband. But you know what, we were attracted to each other, and that is what counts.
But I have looked at many plainer men, and found them totally hot. When I was younger, I was totally hot for the actor Richard Dreyfus, who was far from the sexiest man alive.
The movie Alfie is a good example Alfie and Ruby (Shelly Winters) were very different yet he said she was the one he loved the most.