I don't get that because of my physical size (as I'm not that big), but I get it for my looks a lot. When a lot of people see me, they see my hairstyle, piercings, tattoos, clothes, and assume I'm a loser / street trash.
When I was dating my last boyfriend, I went with him to a party with his friends. I recognized one person at the party that I used to hang out with, and most of the rest of the people there I didn't really care for. Weren't really my type of people; they seemed really judgmental, and were gossiping about other people at the party. "Did you see who ___ brought with them? Omigawd, I can't believe ___ wore that tonight. SOMEBODY gained weight!" I was bored out of my mind, and hung out with the person I knew while my bf made his rounds to mingle. Later that night, I kept hearing a small group of guys not too far away from me saying "rough trade" and giggling a lot. I heard it a few more times, and walked away. My friend came to me not much longer after that, and asked me to hang out with him elsewhere at the party, because these guys were saying I was rough trade. So I went to the group of guys and introduced myself, and after a bit of small-talk, they started asking how I liked to have sex, etc. After being hit on a bit, and telling them how pointless their lives seemed to me, I decided to take my own tour of the house. I found a bathroom upstairs, and decided to leave them a gift. I upper-decked (took a shit in the take of, rather than the bowl of) their toilet, told my boyfriend that I didn't feel comfortable around those people, and left. I never told the boyfriend what I did, but heard him on the phone a couple days later with his friend, asking his friend "That's disgusting, who would do something like that?" Perma-smile! Hahaha.
As for other people thinking I'm stupid for how I look, I see it as an advantage. When people judge my character based on having facial piercings or a hairstyle they don't like, it does a nice job of weeding out people I wouldn't want to waste my time getting to know anyway. People don't need to know I've got a great job, a rockin' GPA, pay for a lot of my tuition through scholarships based on SAT scores and scholastic testing, and what my life passions are. I save those conversations for people who can see past my physical appearances. Anyone judging you based on your looks probably won't listen to what you have to say anyway. They've already got their own opinions of you based on their stereotype.