Yeah, Drug use is just an attribute.. something you have to decide if you want to accept, live with it (neutral) or reject it.
Drug abuse is more of an issue about pathology and its pretty much a valence issue that no one wants to be involved with someone who is abusing, has a psychotic disorder etc., unless they share some equal or similar trait.
I don't think there is anything wrong with doing drugs, but I think that its silly to kid yourself that its something more than trying to reward your brain and feel euphoric.
I think I would have to understand a persons motives for using drugs before I considered being in a relationship with them. The fact of the matter is that drug use is correlated to a lot of negative behavior and other pathologies that I would not find attractive