For me, it will always be about what feels comfortable to write about. I can't set a story in a part of the world I've not experienced, or in a period either. Plenty can and do, so those who enjoy that aspect don't miss out.
Kink-wise, my thoughts are the same. I can only write honestly about things I've experienced (even if disastrously). The exception that proves the rule is the odd story I've set in the future, as that is creative speculation, not research- based. I would never suggest this is a case of right or wrong, but 'feels right or wrong for me to do' is an entirely different standpoint.
If people's enjoyment of a story is genuinely affected by a historical inaccuracy, or seen as impossible because of geographical or even physical improbability errors, they are perfectly entitled to take that view, just as I am to choose the settings and activity I do.
Doing what I do for fun allows me to stay in my lane. What seems restricting to some might feel like an important safety net to others. Not being interested in chasing badges or wanting to 'push boundaries' can be very liberating! If I need to Google it, except to confirm I'm correct, I wouldn't be happy writing about it...but that's just me.