First, free speech is about the perception of power imbalances. If person or group A is perceived to have power over person or group B, and A blocks B from expressing something, and it is believed that A did this because A disapproved of what B was trying to express, A will be be accused of “censoring” B.

The precise nature of the power involved is not generally relevant; common examples of A include the government, large corporations, schools, list and board moderators, and majorities of voting-based discussion sites (Slashdot, Reddit, HN, SO, etc.).

Basically, there seems to be a feeling that people with control over any discussion have a duty to allow all viewpoints to take part. Perhaps some connections can be drawn to natural justice and “hear the other side” (audi alteram partem)….

Second, people are usually OK with private actors censoring speech that goes directly against their interests and uses their own resources–it’s speech on issues orthogonal to their mission that people feel needs to be protected.

Roughly, the first type can be thought of as “not allowing advertisements for the competition in your own store”. Some things that should generally be OK: banning atheists from religion forums; banning Macintosh evangelists from Windows forums; banning creationists from science forums. What should not is banning people from technical forums over their real-world political affiliations and opinions.

To cite a recent, prominent example, I think most people would have been fine with forcing Brendan Eich out of his job over support for Prop. 8 if he’d been working for the ACLU or GLADD, or fine with forcing him out of Mozilla if he were secretly supporting Internet Explorer somehow.

(Criticism of the leadership of the venue hosting the criticism may be something of a special case, and I haven’t really thought much about it yet. I think allowing people to disagree with your decisions on your own forum may take the most commitment to free speech of all, but on the flip side, deleting those threads is probably the fastest way to get your entire userbase to hate you.)