A 'gay man' is still physically male. Unless you meant 'man' as in a 'masculine man'. The concept of masculinity and femininity is a social illusion, important, yes, but still an illusion.
What's the difference between a 'man' and a 'gay man' for instance, in your opinion? For that matter what's the difference between a 'man' and a 'woman' disregarding the obvious physical differences?
Whatever your answer to that, it always will be one thing: it's artificial. All social gender roles are. Everything from wearing skirts vs. pants, to watching football vs. watching soap operas.
And no, being gay isn't simply about sex, and you should know that. It encompasses everything in terms of attraction including romantic and emotional bonds. A 'man happening to like men' is still gay. Perhaps you're mislead by the '-sexual' in 'homosexual' and equate gayness with only sex. To compare it with straight men is not quite applicable either, because straight men are also just as defined by their heterosexuality as gay men are by theirs. Perhaps even more so, that they have to stereotype homosexuals, bisexuals, and transgenders, and even women just so they will feel safer inside the boundaries of their own ideas of what a 'straight man' should be. Misogyny and homophobia are very closely linked to the 'masculine image' that heterosexual males always want to live up to.
That said if you meant 'gay' as in the cultural aspect of gayness, then yes some of us are into it, some of us aren't into it, some are apathetic, some are rabid on gay issues. But THAT, isn't necessarily sexual. It's a struggle for the recognition that our attraction to the same sex is real and just as natural as heterosexual attraction and is not motivated nor defined by sexuality, but by the larger human issue of tolerance and equality.
And no, just because you're gay doesn't mean you listen to purely gay music, or watch only gay shows, gay TV, gay newspapers, gay movies. But most likely you'd gravitate more to them because like it or not, there are differences between you and heterosexuals that simply can't be satisfied elsewhere. You'd gain greater satisfaction, for instance, reading about two men in a love story than a man and a woman. If you do all of those, or some of those, or none of those, so what? You're still a human male.
The constant assertion of being a 'man' as in a 'masculine man' reminds me of g0ys. A 'man' does not mean a 'straight man'.
I'm human first.