Personally, I associate the physical attraction with the sexual aspect of a relationship.
It's kind of like vitamins. In the grand scheme its a very small but critical aspect of starting a romantic relationship (as opposed to a purely platonic one).
Someone with astounding physical beauty to me, can quickly loose attractiveness if the intangible attributes are lacking. Whereas, a guy's who's not 'remarkable' in looks can grow on me easily if there's a small physical attraction initially.
Here's the qualifier though. I'm someone who's more attracted to a good body than a handsome face. The advantage of that is that there are plenty more men out there who can be 'hot' to me. The disadvantage is that of the men that I've met who meet these broad criteria are often of the belief that I really prefer the opposite. I don't understand it other than assuming that it's social conditioning that you have to be 'handsome'.
...sigh...I guess it's one of the reasons that I'm single.