I doubt there was ever a time in America, pre-Revolution and since, that race was not an issue. It was for the framers of the Constitution who, in order to get the southern colonies to accept it, included in Article Two that, for the purpose of taxation, slaves were to be identified as only “three-fifths” of being a person. In Section 9, it was agreed that the issue of slavery was not to be addressed until 1808, but “a tax or duty may be imposed on such importation, not exceeding ten dollars for each person.”
