How people should respond to racists who try to speak on college campuses (such as Charles "blacks are genetically less intelligent than whites" Murray's recent attempt to speak at Middlebury College reported here) is a question that we should answer by figuring out how, in the specific circumstances of the moment, we can best prevent their racist views from gaining support and from being acted upon. The notion of a "right to free speech" has absolutely nothing useful to offer in this effort.
Voltaire's Famous Declaration
Let's start by examining the famous statement attributed (perhaps apochrophally) to Voltaire: "I disagree with what you are saying, but I will defend to the death your right to say it." This is a stupid statement. Here's why. "Saying" is a form of "doing." Whenever two or more human beings do something for a shared goal, it is virtually always the case that it required some previous "saying" to make this "doing" possible. Joint actions require prior communication to motivate the action and formulate how to execute it. "Saying" is a key part of "doing"--often the most important part.
Nobody would say, "I disagree with what you are doing, but I will defend to the death your right to do it" would they?
Consider this act of "saying": a bunch of people are driving a truck around the city and using a bullhorn to say (not do, just say, mind you), "It's time to burn the Jews who are the cause of all our suffering; let's start by assembling at 4th and Vine at 5pm today; bring weapons."
Would you say about these "sayers," "I disagree with what you are saying but I will defend to the death your right to say it"? Wouldn't that amount, under the circumstances, to saying, "I disagree with what you are doing but I will defend to the death your right to do it"?
There is no sharp line between "saying" and "doing." A bogus "right to say" amounts to a "right to do." Saying is often merely the first step in doing.
Yelling Fire in a Crowded Theater
Most people make an exception for the "right to free speech" when it is the proverbial case of "yelling fire in a crowded theater." The reasoning is that the "saying" will clearly cause _immediate_ harm. People who defend the "right to free speech except when it is 'yelling fire in a crowded theater'" are wrong in believing that they have a coherent philosophy. This too is a bogus kind of "right to free speech." Here's why.
First, note that the original use of the phrase "yelling fire in a crowded theater" was when Justice Oliver Wendell Holmes used it in defense of the government's right to prevent the distribution of flyers opposing the military draft for World War I. Holmes argued that the flyers might be persuasive for some people, and this might in turn lead them not to enlist in the military, and this in turn could prevent the "War to End All Wars" from ending all wars. Here's the point.
Even the man who coined the phrase "yelling fire in a crowded theater" knew that "saying" is a form of "doing" and that the immediacy and inevitability and specificity of the causal effect of the "saying" on the "doing" is NOT what is important; what is important is the harmfulness of the possible (not even guaranteed) effect (not that I think anti-war flyers were harmful during WWI, but Justice Holmes certainly did.)
Speech is Part of Waging War
WWI was orchestrated by the oppressive classes of the world against the oppressed (working) classes of the world. This was class war. In a war, each side uses "saying" to strengthen its side and weaken the other side. Winning a war means, among other things, prevailing over the enemy and preventing it from using "saying" to reverse the outcome and prevail over one's own side in the war. Justice Holmes knew this, and we should too.
In the class war between oppressors versus the oppressed, the oppressed should aim to win. This would mean creating a society in which doing ANYTHING ("saying" included) for the purpose of oppressing people would be prevented.
The question is not whether to prevent oppressors from doing and saying, the question is how.
When a person who "says" ideas that strengthen the forces of oppression has at least a certain level of support or friendly neutrality in the general public, it may be the case (it depends on the actual circumstances, not some bogus abstract "right to free speech" principle) that physically stopping this person from speaking will be counter-productive because that would only increase sympathy for this person and for his/her ideas. Sometimes the best way to prevent their pro-oppression views from gaining support and from being acted upon is by doing something different, such as demonstrating disagreement with the ideas or refuting them effectively.