On the face of it, 3 weeks is early. But as AMoonHawk said, it depends how you've spent that time together.
Generally speaking, at 3 weeks there is still a lot of wishful thinking. How well do you really know him? It's like when you see someone who looks hot but is just a little bit too far away for you to make out the details. Your brain fills in what you want to see, and more often than not as you get closer, you start seeing he doesn't look quite like you imagined (that's the: "so far, so good" effect, or "distance distortion factor"). Same thing is true with personality. Until you really get to know someone, you fill in details with what you want to be true. Hopefully it is, but... I know a lot of "honeymoon relationships" that ended with the "honeymoon", usually about 3 weeks (give or take).
On the other hand, it's not like you're about to propose or get married. What does the "boyfriend talk" really entail? That you're not just FWB? That you're exclusive?
Here I agree with Bjorkio. You want to know that you're on the same page (sounds like you both feel it already). Communicating that verbally is a good thing.
Then again, some people here have said they would freak out were that discussed after only 3 weeks. But not you, right? And your guy is like you in many ways, including seeking a relationship? You have the same "agenda"? Then maybe it is the right time.
Perhaps one way to soften the discussion is to ask him if it's too soon? "Hey, I really appreciate you and enjoy all the time we've spent together so far. Everyone thinks we're an old married couple. Can we call ourselves boyfriends or is it still too early?"
You've told him that you're ready for this step, but not being a freak about it, and you've given him an easy out if he thinks its needs a little bit more time.