Here's the thing. Attraction is two fold. There's physical attraction, which is usually the first thing that sparks interest in someone, and emotional, which is based on a whole range of traits and characteristics that vary not just from person to person, but interaction to interaction. In emotional attraction, context is everything. It can be just as instant as the physical, and is often more lasting.
So what the OP has is only one of the 2 present. At this point, it becomes about priorities and compromises. I have friends who prioritize physical. Sometimes that works for them, most of the time it just leads to disappointment, even when you're capable of "being that which you most desire" physically. But the OP is going to have to choose, choose wisely, and hope to avoid hurting anyone's feelings, his own included.
Decisions like this are not fun and frankly are like psychic ulcers.
And for a shoe on the other foot point: just remember that if this guy "gets hot" he might have more options than you if he's got the winning personality and characteristics that you describe. HIS grass may get greener than you.