I think the issue is that people generally like you (which would also be people generally like me, and like most people here), simply can't grasp how different things are for people not like us.
That isn't to say people generally like us don't have or can't have sympathy (because we're generally good people) or even empathy (because white men face adversity also of course). But it does sort of mean, I think, that we should err on the side of listening more than we speak on matters like this.