Mutual attraction is healthy, smart, and requisite for longevity in a committed partnership. You can't put up with the ever expanding list of petty shit that emerges once the bloom is off the rose if you ultimately aren't physically attracted to someone.
However, that is not sufficient reason to not run them over in the driveway (accidentally officer, truly!) the 10,000th time they leave the lid off the peanut butter. You have to have something deeper and complimentary, balance, trust, admiration, curiosity, respect, tenderness and spiritual commonality. You have to like the person as a person, and be able to think of them as the perfect witness to your life, and the person whose life you most want to witness and celebrate.
Physical/sexual attraction and appeal are funky quirky things, and beauty is unique for each of us. We all have our d'ruthers, and if yours is "I'd'ruther be fucking someone else" then you're with the wrong guy. But if you can know him, love him in spite of his insanity, and still be able to objectify him and want to fuck him senseless and then cuddle him...you're not shallow, your damned lucky.
Attraction is real, and feeling guilty about it is a waste.