I can understand what you mean because I get the same kind of thing at times for the way I am. But even if I do get discredited because I'm not the ideal of masculine, I just don't let it bother me. People will judge you for anything, especially in the gay world (in my experience). But at the end of the day, you have to ask yourself. Assuming you are okay with how you are, is it really worth changing yourself just to please criticizers what you want. Me personally, I think not.
Like GTA5 said, do what you like. I'll put myself out there. Why do I wear predominantly black? Because I like it. Why do I wear my hair the way I do? Because I like it. Why do I wear underbust corsets? Because I want to and like them. I understand that a lot of my personal tastes don't scream masculine tough macho man but I don't care. I'm living for myself.
But that's just my 2 cents.