I am not sure why anyone would “fear” being called racist. If you did something and someone says it was a racist thing to do, sit down with yourself and reflect. Listen to why they said it was racist. Maybe you will learn something. Maybe you will conclude the person is wrong. Just don’t automatically get defensive. A person doesn’t have to be a member of the KKK or burn crosses on someone’s lawn to be racist. Racism comes in many forms.
My brother and SIL have a grandchild who has a black father (they are white.) Every time she comes to their house (she is 3), one of the first things my SIL does is takes out her braids and washes and straightens her hair. Oh, and changes her clothes if they are going somewhere. This is racism. With their own grandchild. They are trying to make her as white as they can (she is light skinned.) They would go on a rampage if you pointed it out to them. My niece has yet to confront them about it, but knows she will have to soon as this is very damaging to this young child.