Being big and fit has a number of benefits other than just the big and fit and healthy part: it's a very sexual thing and folks respond at a primal level.
Their reactions can be varied:
As someone who has been a 9.1 to 9.9 on the ratings for years, as someone who holds trophies in bodybuilding and powerlifting as well as ice hockey, and having been active and built all my adult life (my 33rd year of lifting), I've experienced all the above for many years. It is what it is, and a by-product of being "different". I don't feel it's "normal" to be fat, and out of shape, nor real thin, but, facts are that most folks are kind of lazy, don't eat right, and lack discipline.
I wouldn't sweat it. It beats chemotherapy.
How do I take it? With a grain of salt. There's always someone bigger, or better looking, or more fit, younger, etc. There's always the haters. There's always the skinny guys that think I should be attracted to them because they adore me. There's always the young guy who won't eat who thinks I have some magic pill. There are worse things in life.