Were they drinking when they gave out their number? And how long did you wait to call?
I don't mind if someone calls me the next day, as long as they don't call me constantly. I have a really busy schedule, and will get back to someone when I can. If someone calls me 3 weeks later, however, I may not call them back. If I've been drinking heavily, which generally increases my chances of flirting and giving out my number, I may have only been interested that night, while I was buzzed.
There's really no rule to how long you should wait before you call someone. The whole 'wait 3 days' thing is bullshit. If it takes you 3 days to get back to someone, that's fine. If you have time to call them 2 days later, I don't think they're going to freak out about it, thinking you should have waited. Just pick up the phone and call. If they don't answer and don't return your call, they're just not interested (anymore?). No big deal.