When you've always had privilege, equality can feel like a loss