I've been in public men's restrooms when women will come in to use a stall, often explaining that there's a waiting line at the women's. I've never heard a man complain, though a few of us exchanged some surprised glances. And I would fully expect that in many locales, a man who did the same thing in the ladies restroom would be arrested, Tennessee notwithstanding.
In fact, one time in Mall of America at Minneapolis, the largest shopping mall in the US, I quickly rushed into a large restroom that had no door, just a maze entrance like you see in airports. I couldn't see where the urinals were, but in my urgency I just went into a stall.
A moment later I heard several female voices, and immediately realized I must have gone into the ladies restroom by mistake. What to do? Wait until they left, or would more women enter, keeping me trapped in that stall indefinitely? And what if one of them noticed my man's shoes & pants, visible under the stall door & partition?
I zipped up, took a deep breath, and threw the stall door open, saying as I raced for the exit: "Sorry ladies, I got the wrong one!"
I kept going at a jog, fearing the Mall police might have been alerted, not stopping until I reached the Rainforest Cafe, if you know it there. I ducked into their bar, and had 2 gin martinis in quick succession until I had calmed down. I wasn't arrested.
Another place in Minneapolis-St Paul was a lesbian club called the Metro, now closed. They had exclusive women's restrooms for the ladies, but only a Unisex for the men. It was the first time I had to pee in front of women in a public place.
But I must admit I've also been in gay clubs that had no separate restroom for women, who had to use the men's along with the rest of us. Of course it's often difficult for me to know if the ladies in there are really females, or drag queens, or trannies, or what.