This phrase has never sat well with me. I understand its intended meaning: a Person of Color’s race, ethnicity, and/or culture does not negatively impact my perception of them.
Oh, but it does.
The origin of the term, so far as I know, extends to the famous Plessy v Ferguson Supreme Court case: Justice John Marshall Harlan wrote a blistering dissent in which he said, among other things, that the Constitution “should be color blind.” But Harlan was born into a prominent slaveholding family in the antebellum south: as such he was likely saying something much closer to, ‘we recognize the problem of color, but choose to overlook it,’ in a sort of patronizing magnanimity. It’s the white mans version of ‘noblesse oblige.’