It mostly seems to be
Women who are quite mentally ill
Women who are bad parents (if they have kids)
Women who are banging in bed
Women who are tall. I'm 6'1 and quite broad and they tell me it makes them feel feminine

If I ever get to the point where I can't get hard it wouldn't bother me as much as if I was told I could never go down on a woman again. I love giving oral to a woman.
TBF if you get to the point that you can't get hard then oral sex really is the thing. Having someone slide their lips up and down your shaft feels good whether your hard or not in my experience. Just make sure your oral sex game is on point so you can return the favour
As for the original post I would say that it's wise to communicate with people and tell them what you want and expect. It's definitely not your fault either and refusing to go down on them might seem an appropriate response but it could escalate to the most minimal unsatisfying sex and then you both lose.