Indeed. It isn't a new thing.

I worked in a place a long while ago, being the only male in that department.

Everyone was nice and we did our jobs.

Over time I discovered 'rumors' about me from people outside the department. I'll call these people, the observers. They had no interactions with our team, aside from seeing who worked in our office as they walked past.

It was quite concerning that people would assume the situation meant anything but myself working with a team of peers.