I am not sure it is anyone's place to determine whether one should be offended or not because of differing cultural contexts.
Muhammed Ali, Harry Connick Jr and Adam Goodes were all offended at what happened. Is it not enough to leave it at that? Are we now the moral arbiters of who can be offended by what?
If it were left at that, there would be no issue.
I think much of the conflict arises from those purporting to get in the heads of others and ascribe a racist mentality to them.
In cases where the words or actions are not clearly objectively racist, that can be a harsh label to tag somebody with.