A sitter touching you intimately is sexual child abuse. Not a good introduction for you to activities that later, among other consenting adults, you should have found to be normal, healthy & enjoyable.
We are from the same era (born 1949). My own childhood approach to sexuality was a bit like yours, yet also different, that only included one creepy moment with an old guy. Like you, for instance, I wasn't focused on my penis, except to learn there were rules about when I could show it, and when it had to be covered. And also like you, that my dick had a "dirty" aspect to it.
But its sexual potential was unknown to me, just the thing you peed with, and didn't think about it otherwise. As for adult men, I didn't think about theirs, either. Examples:
- When I was fairly young I might be tagging after my father doing his yard work, our place surrounded by woods then, with few neighbors. If either of us had to pee we'd do it outdoors, and often literally have a "pissing contest" for distance and accuracy. You know, I never looked at my father's organ, no interest, didn't even realize it had hair there, and to this day never knew if he was cut or uncut. We were just peeing, what all men & boys did in company, not sexual at all.
- I swam naked in our backyard pool until about 9 or so, usually with either Mom or Dad (suited) present for safety (my topless younger sister sometimes in the water, too). Yet I knew I had to wear a swimsuit when going to the public beach, where that same nudity would be shameful.
- At summer camp beginning at 8 I'd go overnight group camping in the woods, and we'd skinny-dip in the creeks. Our college-age counselors were naked, too, and the sight of their bushy pubic hair and larger stuff was a shock to me and some of the other boys. By 9 I was starting to find the sight more attractive, no longer for its mere novelty. And once one of them popped a boner he couldn't lose for a few minutes, and that was really an eye-opener for most of us boys, who knew little about that male ability. Though I remember some boys already knew enough about them (I didn't), to use the terms hardon and boner.
- Naked open swims at the Y were segregated by age, only boys below 16 during one period, and those 16 & up with the adult men at a different time. (Swim classes were more narrowly age-defined) But I do remember beginning to pay attention to younger teens sprouting hair, and how the other boys reacted to it.
- Reaching my own teens I became very self-conscious & shy, mostly because I matured very early, making me different, and I was hung like a horse. I was afraid the other guys would think I was having a hardon, even when I was soft, plus I was always at risk of getting an actual one in the showers or pool. I didn't understand my new sexuality, and that the other naked boys were turning me on. I avoided those situations, no more Y swimming. It wasn't until the US Army that I was forced to finally acquire more self-control, living in situations where I couldn't avoid male communal nudity.
- Gay sex had to wait until my 40s, but when it happened it was glorious, best sex of my life (straight sex had been an awkward & uninspiring experience for me). I love gay sex, no shame, no remorse, no guilt, only regret that it took me so fucking long to begin it.
I hope this gives you some additional perspective, from someone your own age.