I say it to a man I've been dating when I'm certain that I can honor it with my heart, not just my dick (sorry, but being blunt) and when I am certain that nobody else would have me wondering if I should have said so. I have, once before, spoken much too quickly and it was a painful mistake.
My last partner of 16 years and I said it to each other within days of meeting. (That was not too soon, in that instance, it was very true.) It's still true, but its context has shifted to something unique, which brings us to
Friends - I say it when I realize that if I were without them it would all be a little darker and more lonely, and that I cannot let the moment pass, for fear they may not hear it.
As far as Family goes, I grew up hearing it, sharing it, and saying it with my family, and I say it again when I feel time racing by me, or when I realize that their stupidity and insanity still isn't a deal breaker.
I say it to the dead, each time I meditate on them or visit their grave.
And always -- and this is going to upset or confuse some folks, I know - but always it seems to me, particularly as I get older, that saying "I love you" is half spoken out loud to the the person I love, and half whispered to God as a prayer of thanks.