OW is right. There was actually a great story in the NYT about a study that was done concerning doctors who either A) don't listen to their patients and/or B) spend more time talking about themselves.
The whole article was unnerving, particularly given the state of our healthcare system. You are faced with another challenge, Gigaram, in that the northeast (particularly MA) is experiencing a great shortage in primary care physicians. You may find a good one, and they are either not accepting new patients, or some have a 7 WEEK wait time. Absolutely unacceptable in my mind.
Above all, gender doesn't matter beyond personal preference. Competency, compassion and someone who will hear you out about your concerns should be the only things you focus on. If you are more comfortable around one sex, then follow your feelings. The last thing you'd ever want to do is not report something to your physician because you aren't comfortable being open and honest with them.
The physician/patient relationship SHOULD be a relationship. You sit down, discuss issues, collaboratively come up with a treatment plan and implement it. The whole HMO-mandated "you must see X # of patients a day" crap is severely cutting into the quality of health care.