It sounds as though you have trouble setting what psychologists call "boundaries." I do too, especially with my ex-partners. It appears you're no longer physically attracted or emotionally connected with your ex's. If that's the case, be blunt. Tell them you're not interested and remind them how inappropriate late-night phone calls and unannounced visits to your place are. If they persist, take your phone off the hook at bedtime, and don't answer the door when they arrive at your home.
On the other hand, if you're still physically attracted to them, enjoy a no-strings night of sexual frollicking.
I feel somewhat hypocrital offering that advice because I've only had mixed luck dealing with two ex-partners I have lost ALL physical attraction to. One of them has a major muscle fetish, and I find it virtually impossible to refuse his requests, which could be more accurately described as begging, for me to do cheesy bodybuilder poses (which I loathe because they make me feel like a narcissist and an exhibitionist) while he masturbates in my bed. We don't have any physical contact, but the scene still creeps me out. I also find it hard to turn him down because he is always so grateful...plus he fixes my computer for free. LOL
I've had better luck setting boundaries with another ex, although not until enduring three grotesque encounters with him, which if they had taken place between strangers would amount to sexual assault. This ex is a brilliant conversationalist, and I have maintained a platonic relationship with him for that reason, even though my sexual attraction to him has evaporated. Until recently, it was easy to keep the relationship platonic because he is in a mongamous relationship - or so he claimed.
But on three separate occasions recently, he showed up at my door unannounced. While we were sitting on the couch in my living room, having a nice chat about Constitutional law (he's an attorney), he suddenly ripped off my shirt and jumped on top of me. He's a skinny geek, and I teach martial arts, so I didn't feel in any danger because I could have easily stopped the assault. Instead, I just sat there motionless until he got bored by my lack of reciprocity, and then we returned to chatting about whether or not the First Amendment guarantees private gun ownership.
This happened two more times, and I finally had the long overdue epiphany that his behavior demonstrated a grotesque lack of respect. When I asked him how he reconciled his promise to be monogamous with his current lover (for whom he had dumped me!) with his sexual overtures toward me, his nonsensical response was, "I was just flirting with you." I told him that if he had done that to a woman, she would have had him arrested for attempted rape.
As much as I had enjoyed chatting with him about whether school vouchers violate the First Amendment's separation of church and state, et al., I finally read the writing on the wall, which said, "You lack all self-respect if you continue to let this guy use you as his private sexual buffet." And so I regretfully ended all contact with him, although I still miss our discussions about whether King George II's untreated alcoholism explains the madness of his adventure in Iraq.