As Jake says: "I hate Illinois Nazis". I also hate Missouri Nazis. I hate Wisconsin Nazis. I hate Nazis from all over.
On the one hand, we have a set of values and standards calling for peaceful resolution of disputes and tolerance for alternative, even if despicable, ideas. If we start responding to speech with violence, we become just like them.
On the other hand, they have a set of values and standards different from ours. If you don't respond to their speech with violence, they won't understand, and will think you are weak. Their entire strategy is to exploit the weakness of the system to gain power, and to change the system to retain power.
The second hand is correct. You really need to think about two groups, we, the civil, and they, the uncivil. You treat the civil with civility, that is how civilization was built. You treat the uncivil with barbarity, that is how civilization is preserved.
We in the US have a Constitution, as imperfect and imperfectly administered as it has been historically, which governs individual and government rights and roles. If you subscribe to the Constitution, you are eligible to receive its protections (being secure in life, liberty, and property and so on, like not being punched in the face, and having the puncher jailed). You are a member of Team Civilization. If you do not subscribe to the Constitution and the system it engenders, you are not eligible to receive its protections.
Now, who is to decide who are the good guys and who are the bad guys? Each and every one of us does this daily, at our own risk. There is ambiguity, some of our fellow citizens obviously miscalculate. Almost everyone thinks they are the good guys. If we all assessed this correctly, or at least identically, politics would be much simpler.
Here is a brief 2x2 matrix describing behavior: Who is in Power? Nazis Team Civilization Who is acting Nazis violent until done crushing violent until in power Team Civilization violent until crushed or victorious generally non-violent, except with Nazis So the question arises: Is some neo-Nazi, like say Richard Spencer, violent. His beliefs are not something I know at a personal level. However, he appears to be trying to make the US a white ethno-state. This means he is essentially calling for the "peaceful" expulsion of non-whites (as defined by him). I argue that he is advocating (seemingly through peaceful means) violence. This gets to the question of what is voluntary and peaceful.
At the risk of opening a philosophic can of worms, interaction can considered on a continuum between voluntary and compulsory. Free exchange between two parties is in some respects voluntary. However, if the good is necessary, like food, then while the particular exchange may be voluntary, the underlying demand is compulsory. A monopolist would have existential power. The archetype of compulsory relationships, the gun to the head as an implied threat of death when offered the opportunity to freely hand over your money, has in it some voluntary elements - though the “choice” of attempting disarmament or dying may not be appealing compared with turning over your money. The gunman has far more freedom of action than the victim in this case.
If in Spencer's America, non-whites don't voluntarily leave after some time window has passed, surely they will be forced to leave by the state, the institution with a legal monopoly of force. The state is the gunman in our "voluntary" expulsion. Calling for a white ethno-state is advocacy of violence, even if you are calling for it to be "legal" and "peaceful" and"constitutional".
Saying "I wish you were dead" is not a crime. Acting to speed along someone's death is.
In contrast, as far as I can tell, aside from neo-Nazis of various stripes (or should I say swastikas), who are clearly a minority, and their confederates, the vast share of the remainder of the US is not calling for expulsion or extermination of fellow citizens. We could go deeper and look at whether and which non-citizens are allowed to immigrate to the US. As we know, due to Prospect Theory, a loss of something you have (expulsion) is considered far worse than not winning (being prohibited from entering), even if both result in you being in the same place. Team Civilization gets to decide its membership. I personally think the membership should be broader, but the Constitution establishes the process by which the rules of membership are established. I will voice my opinion, but will not violently rebel.
Though I generally wouldn't do it personally, it's morally acceptable for individual members of Team Civilization to punch Nazis. It is proactive self-defense. Your body does this all the time with the immune system. It probably should not be legal, nor officially sanctioned by the state, as many members (at least 47%) of Team Civilization appear to have poor judgment. But the punishment should fit the crime. However, if you mistake a member of Team Civilization for a Nazi when punching them, you should go to jail. We cannot tolerate auto-immune diseases. As a wise man once said: "Aim for the king best not miss."
As has been pointed out, we went to war with the Fascists (and they did start it) and didn't merely punch them, we killed them, which is objectively worse.
There are other groups which advocate violence through nominally peaceful means. They need to be treated with similar immune system defenses.
Individuals who act violently are already treated this way.
In short, the only thing we cannot tolerate is intolerance.