by Julia Angwin, ProPublica, and Hannes Grassegger, special to ProPublica
In the wake of a terrorist attack in London earlier this month, a U.S. congressman wrote a Facebook post in which he called for the slaughter of "radicalized" Muslims. "Hunt them, identify them, and kill them," declared U.S. Rep. Clay Higgins, a Louisiana Republican. "Kill them all. For the sake of all that is good and righteous. Kill them all."
Higgins' plea for violent revenge went untouched by Facebook workers who scour the social network deleting offensive speech.
But a May posting on Facebook by Boston poet and Black Lives Matter activist Didi Delgado drew a different response.
"All white people are racist. Start from this reference point, or you've already failed," Delgado wrote. The post was removed and her Facebook account was disabled for seven days.
A trove of internal documents reviewed by ProPublica sheds new light on the secret guidelines that Facebook's censors use to distinguish between hate speech and legitimate political expression. The documents reveal the rationale behind seemingly inconsistent decisions.
For instance, Higgins' incitement to violence passed muster because it targeted a specific sub-group of Muslims — those that are "radicalized" — while Delgado's post was deleted for attacking whites in general.
Over the past decade, the company has developed hundreds of rules, drawing elaborate distinctions between what should and shouldn't be allowed, in an effort to make the site a safe place for its nearly 2 billion users.
The issue of how Facebook monitors this content has become increasingly prominent in recent months, with the rise of "fake news" — fabricated stories that circulated on Facebook like "Pope Francis Shocks the World, Endorses Donald Trump For President, Releases Statement" — and growing concern that terrorists are using social media for recruitment.
While Facebook was credited during the 2010-2011 "Arab Spring" with facilitating uprisings against authoritarian regimes, the documents suggest that, at least in some instances, the company's hate-speech rules tend to favor elites and governments over grassroots activists and racial minorities.
In so doing, they serve the business interests of the global company, which relies on national governments not to block its service to their citizens.
One Facebook rule, which is cited in the documents but that the company said is no longer in effect, banned posts that praise the use of "violence to resist occupation of an internationally recognized state."
The company's workforce of human censors, known as content reviewers, has deleted posts by activists and journalists in disputed territories such as Palestine, Kashmir, Crimea and Western Sahara.
One document trains content reviewers on how to apply the company's global hate speech algorithm. The slide identifies three groups: female drivers, black children and white men. It asks: Which group is protected from hate speech? The correct answer: white men.
The reason is that Facebook deletes curses, slurs, calls for violence and several other types of attacks only when they are directed at "protected categories"—based on race, sex, gender identity, religious affiliation, national origin, ethnicity, sexual orientation and serious disability/disease.
It gives users broader latitude when they write about "subsets" of protected categories. White men are considered a group because both traits are protected, while female drivers and black children, like radicalized Muslims, are subsets, because one of their characteristics is not protected. (The exact rules are in the slide show below.)
The Facebook Rules
Facebook has used these rules to train its "content reviewers" to decide whether to delete or allow posts. Facebook says the exact wording of its rules may have changed slightly in more recent versions. ProPublica recreated the slides.Behind this seemingly arcane distinction lies a broader philosophy. Unlike American law, which permits preferences such as affirmative action for racial minorities and women for the sake of diversity or redressing discrimination, Facebook's algorithm is designed to defend all races and genders equally.
"Sadly," the rules are "incorporating this color-blindness idea which is not in the spirit of why we have equal protection," said Danielle Citron, a law professor and expert on information privacy at the University of Maryland. This approach, she added, will "protect the people who least need it and take it away from those who really need it."
But Facebook says its goal is different — to apply consistent standards worldwide. "The policies do not always lead to perfect outcomes," said Monika Bickert, head of global policy management at Facebook. "That is the reality of having policies that apply to a global community where people around the world are going to have very different ideas about what is OK to share."
Facebook's rules constitute a legal world of their own. They stand in sharp contrast to the United States' First Amendment protections of free speech, which courts have interpreted to allow exactly the sort of speech and writing censored by the company's hate speech algorithm. But they also differ — for example, in permitting postings that deny the Holocaust — from more restrictive European standards.
The company has long had programs to remove obviously offensive material like child pornography from its stream of images and commentary. Recent articles in the Guardian and Süddeutsche Zeitung have detailed the difficult choices that Facebook faces regarding whether to delete posts containing graphic violence, child abuse, revenge porn and self-mutilation.
The challenge of policing political expression is even more complex. The documents reviewed by ProPublica indicate, for example, that Donald Trump's posts about his campaign proposal to ban Muslim immigration to the United States violated the company's written policies against "calls for exclusion" of a protected group.
As The Wall Street Journal reported last year, Facebook exempted Trump's statements from its policies at the order of Mark Zuckerberg, the company's founder and chief executive.
The company recently pledged to nearly double its army of censors to 7,500, up from 4,500, in response to criticism of a video posting of a murder.
Their work amounts to what may well be the most far-reaching global censorship operation in history. It is also the least accountable: Facebook does not publish the rules it uses to determine what content to allow and what to delete.
Users whose posts are removed are not usually told what rule they have broken, and they cannot generally appeal Facebook's decision. Appeals are currently only available to people whose profile, group or page is removed.
The company has begun exploring adding an appeals process for people who have individual pieces of content deleted, according to Bickert. "I'll be the first to say that we're not perfect every time," she said.
Facebook is not required by U.S. law to censor content. A 1996 federal law gave most tech companies, including Facebook, legal immunity for the content users post on their services.
The law, section 230 of the Telecommunications Act, was passed after Prodigy was sued and held liable for defamation for a post written by a user on a computer message board.
The law freed up online publishers to host online forums without having to legally vet each piece of content before posting it, the way that a news outlet would evaluate an article before publishing it.
But early tech companies soon realized that they still needed to supervise their chat rooms to prevent bullying and abuse that could drive away users.
America Online convinced thousands of volunteers to police its chat rooms in exchange for free access to its service. But as more of the world connected to the internet, the job of policing became more difficult and companies started hiring workers to focus on it exclusively. Thus the job of content moderator — now often called content reviewer — was born.
In 2004, attorney Nicole Wong joined Google and persuaded the company to hire its first-ever team of reviewers, who responded to complaints and reported to the legal department. Google needed "a rational set of policies and people who were trained to handle requests," for its online forum called Groups, she said.
Google's purchase of YouTube in 2006 made deciding what content was appropriate even more urgent. "Because it was visual, it was universal," Wong said.
While Google wanted to be as permissive as possible, she said, it soon had to contend with controversies such as a video mocking the King of Thailand, which violated Thailand's laws against insulting the king.
Wong visited Thailand and was impressed by the nation's reverence for its monarch, so she reluctantly agreed to block the video — but only for computers located in Thailand.
Since then, selectively banning content by geography — called "geo-blocking" — has become a more common request from governments. "I don't love traveling this road of geo-blocking," Wong said, but "it's ended up being a decision that allows companies like Google to operate in a lot of different places."
For social networks like Facebook, however, geo-blocking is difficult because of the way posts are shared with friends across national boundaries.
If Facebook geo-blocks a user's post, it would only appear in the news feeds of friends who live in countries where the geo-blocking prohibition doesn't apply. That can make international conversations frustrating, with bits of the exchange hidden from some participants.
As a result, Facebook has long tried to avoid using geography-specific rules when possible, according to people familiar with the company's thinking. However, it does geo-block in some instances, such as when it complied with a request from France to restrict access within its borders to a photo taken after the Nov. 13, 2015, terrorist attack at the Bataclan concert hall in Paris.
Bickert said Facebook takes into consideration the laws in countries where it operates, but doesn't always remove content at a government's request.
"If there is something that violates a country's law but does not violate our standards," Bickert said, "we look at who is making that request: Is it the appropriate authority? Then we check to see if it actually violates the law. Sometimes we will make that content unavailable in that country only."
Facebook's goal is to create global rules. "We want to make sure that people are able to communicate in a borderless way," Bickert said.
Founded in 2004, Facebook began as a social network for college students. As it spread beyond campus, Facebook began to use content moderation as a way to compete with the other leading social network of that era, MySpace.
MySpace had positioned itself as the nightclub of the social networking world, offering profile pages that users could decorate with online glitter, colorful layouts and streaming music. It didn't require members to provide their real names and was home to plenty of nude and scantily clad photographs.
And it was being investigated by law-enforcement agents across the country who worried it was being used by sexual predators to prey on children. (In a settlement with 49 state attorneys general, MySpace later agreed to strengthen protections for younger users.)
By comparison, Facebook was the buttoned-down Ivy League social network — all cool grays and blues. Real names and university affiliations were required. Chris Kelly, who joined Facebook in 2005 and was its first general counsel, said he wanted to make sure Facebook didn't end up in law enforcement's crosshairs, like MySpace.
"We were really aggressive about saying we are a no-nudity platform," he said.
The company also began to tackle hate speech. "We drew some difficult lines while I was there — Holocaust denial being the most prominent," Kelly said.
After an internal debate, the company decided to allow Holocaust denials but reaffirmed its ban on group-based bias, which included anti-Semitism. Since Holocaust denial and anti-Semitism frequently went together, he said, the perpetrators were often suspended regardless.
"I've always been a pragmatist on this stuff," said Kelly, who left Facebook in 2010. "Even if you take the most extreme First Amendment positions, there are still limits on speech."
By 2008, the company had begun expanding internationally but its censorship rulebook was still just a single page with a list of material to be excised, such as images of nudity and Hitler. "At the bottom of the page it said, ‘Take down anything else that makes you feel uncomfortable,'" said Dave Willner, who joined Facebook's content team that year.
Willner, who reviewed about 15,000 photos a day, soon found the rules were not rigorous enough. He and some colleagues worked to develop a coherent philosophy underpinning the rules, while refining the rules themselves. Soon he was promoted to head the content policy team.
By the time he left Facebook in 2013, Willner had shepherded a 15,000-word rulebook that remains the basis for many of Facebook's content standards today.
"There is no path that makes people happy," Willner said. "All the rules are mildly upsetting." Because of the volume of decisions — many millions per day — the approach is "more utilitarian than we are used to in our justice system," he said. "It's fundamentally not rights-oriented."
Willner's then-boss, Jud Hoffman, who has since left Facebook, said that the rules were based on Facebook's mission of "making the world more open and connected." Openness implies a bias toward allowing people to write or post what they want, he said.
But Hoffman said the team also relied on the principle of harm articulated by John Stuart Mill, a 19th-century English political philosopher. It states "that the only purpose for which power can be rightfully exercised over any member of a civilized community, against his will, is to prevent harm to others."
That led to the development of Facebook's "credible threat" standard, which bans posts that describe specific actions that could threaten others, but allows threats that are not likely to be carried out.
Eventually, however, Hoffman said "we found that limiting it to physical harm wasn't sufficient, so we started exploring how free expression societies deal with this."
The rules developed considerable nuance. There is a ban against pictures of Pepe the Frog, a cartoon character often used by "alt-right" white supremacists to perpetrate racist memes, but swastikas are allowed under a rule that permits the "display [of] hate symbols for political messaging."
In the documents examined by ProPublica, which are used to train content reviewers, this rule is illustrated with a picture of Facebook founder Mark Zuckerberg that has been manipulated to apply a swastika to his sleeve.
The documents state that Facebook relies, in part, on the U.S. State Department's list of designated terrorist organizations, which includes groups such as al-Qaida, the Taliban and Boko Haram.
But not all groups deemed terrorist by one country or another are included: A recent investigation by the Pakistan newspaper Dawn found that 41 of the 64 terrorist groups banned in Pakistan were operational on Facebook.
There is also a secret list, referred to but not included in the documents, of groups designated as hate organizations that are banned from Facebook. That list apparently doesn't include many Holocaust denial and white supremacist sites that are up on Facebook to this day, such as a group called "Alt-Reich Nation." A member of that group was recently charged with murdering a black college student in Maryland.
As the rules have multiplied, so have exceptions to them. Facebook's decision not to protect subsets of protected groups arose because some subgroups such as "female drivers" didn't seem especially sensitive. The default position was to allow free speech, according to a person familiar with the decision-making.
After the wave of Syrian immigrants began arriving in Europe, Facebook added a special "quasi-protected" category for migrants, according to the documents.
They are only protected against calls for violence and dehumanizing generalizations, but not against calls for exclusion and degrading generalizations that are not dehumanizing.
So, according to one document, migrants can be referred to as "filthy" but not called "filth." They cannot be likened to filth or disease "when the comparison is in the noun form," the document explains.
Facebook also added an exception to its ban against advocating for anyone to be sent to a concentration camp. "Nazis should be sent to a concentration camp," is allowed, the documents state, because Nazis themselves are a hate group.
The rule against posts that support violent resistance against a foreign occupier was developed because "we didn't want to be in a position of deciding who is a freedom fighter," Willner said.
Facebook has since dropped the provision and revised its definition of terrorism to include nongovernmental organizations that carry out premeditated violence "to achieve a political, religious or ideological aim," according to a person familiar with the rules.
The Facebook policy appears to have had repercussions in many of the at least two dozen disputed territories around the world. When Russia occupied Crimea in March 2014, many Ukrainians experienced a surge in Facebook banning posts and suspending profiles.
Facebook's director of policy for the region, Thomas Myrup Kristensen, acknowledged at the time that it "found a small number of accounts where we had incorrectly removed content. In each case, this was due to language that appeared to be hate speech but was being used in an ironic way. In these cases, we have restored the content."
Katerina Zolotareva, 34, a Kiev-based Ukrainian working in communications, has been blocked so often that she runs four accounts under her name. Although she supported the "Euromaidan" protests in February 2014 that antagonized Russia, spurring its military intervention in Crimea, she doesn't believe that Facebook took sides in the conflict. "There is war in almost every field of Ukrainian life," she says, "and when war starts, it also starts on Facebook."
In Western Sahara, a disputed territory occupied by Morocco, a group of journalists called Equipe Media say their account was disabled by Facebook, their primary way to reach the outside world. They had to open a new account, which remains active.
"We feel we have never posted anything against any law," said Mohammed Mayarah, the group's general coordinator. "We are a group of media activists. We have the aim to break the Moroccan media blockade imposed since it invaded and occupied Western Sahara."
In Israel, which captured territory from its neighbors in a 1967 war and has occupied it since, Palestinian groups are blocked so often that they have their own hashtag, #FbCensorsPalestine, for it.
Last year, for instance, Facebook blocked the accounts of several editors for two leading Palestinian media outlets from the West Bank — Quds News Network and Sheebab News Agency. After a couple of days, Facebook apologized and un-blocked the journalists' accounts. Earlier this year, Facebook blocked the account of Fatah, the Palestinian Authority's ruling party — then un-blocked it and apologized.
Last year India cracked down on protesters in Kashmir, shooting pellet guns at them and shutting off cellphone service. Local insurgents are seeking autonomy for Kashmir, which is also caught in a territorial tussle between India and Pakistan. Posts of Kashmir activists were being deleted, and members of a group called the Kashmir Solidarity Network found that all of their Facebook accounts had been blocked on the same day.
Ather Zia, a member of the network and a professor of anthropology at the University of Northern Colorado, said that Facebook restored her account without explanation after two weeks. "We do not trust Facebook any more," she said. "I use Facebook, but it's almost this idea that we will be able to create awareness but then we might not be on it for long."
The rules are one thing. How they're applied is another. Bickert said Facebook conducts weekly audits of every single content reviewer's work to ensure that its rules are being followed consistently. But critics say that reviewers, who have to decide on each post within seconds, may vary in both interpretation and vigilance.
Facebook users who don't mince words in criticizing racism and police killings of racial minorities say that their posts are often taken down. Two years ago, Stacey Patton, a journalism professor at historically black Morgan State University in Baltimore, posed a provocative question on her Facebook page.
She asked why "it's not a crime when White freelance vigilantes and agents of ‘the state' are serial killers of unarmed Black people, but when Black people kill each other then we are ‘animals' or ‘criminals.'"
Although it doesn't appear to violate Facebook's policies against hate speech, her post was immediately removed, and her account was disabled for three days. Facebook didn't tell her why.
"My posts get deleted about once a month," said Patton, who often writes about racial issues. She said she also is frequently put in Facebook "jail" — locked out of her account for a period of time after a posting that breaks the rules.
"It's such emotional violence," Patton said. "Particularly as a black person, we're always have these discussions about mass incarceration, and then here's this fiber-optic space where you can express yourself. Then you say something that some anonymous person doesn't like and then you're in ‘jail.'"
Didi Delgado, whose post stating that "white people are racist" was deleted, has been banned from Facebook so often that she has set up an account on another service called Patreon, where she posts the content that Facebook suppressed.
In May, she deplored the increasingly common Facebook censorship of black activists in an article for Medium titled "Mark Zuckerberg Hates Black People."
Facebook also locked out Leslie Mac, a Michigan resident who runs a service called SafetyPinBox where subscribers contribute financially to "the fight for black liberation," according to her site. Her offense was writing a post stating "White folks. When racism happens in public — YOUR SILENCE IS VIOLENCE."
The post does not appear to violate Facebook's policies. Facebook apologized and restored her account after TechCrunch wrote an article about Mac's punishment. Since then, Mac has written many other outspoken posts. But, "I have not had a single peep from Facebook," she said, while "not a single one of my black female friends who write about race or social justice have not been banned."
"My takeaway from the whole thing is: If you get publicity, they clean it right up," Mac said. Even so, like most of her friends, she maintains a separate Facebook account in case her main account gets blocked again.
Negative publicity has spurred other Facebook turnabouts as well. Consider the example of the iconic news photograph of a young naked girl running from a napalm bomb during the Vietnam War.
Kate Klonick, a Ph.D. candidate at Yale Law School who has spent two years studying censorship operations at tech companies, said the photo had likely been deleted by Facebook thousands of times for violating its ban on nudity.
But last year, Facebook reversed itself after Norway's leading newspaper published a front-page open letter to Zuckerberg accusing him of "abusing his power" by deleting the photo from the newspaper's Facebook account.
Klonick said that while she admires Facebook's dedication to policing content on its website, she fears it is evolving into a place where celebrities, world leaders and other important people "are disproportionately the people who have the power to update the rules."
In December 2015, a month after terrorist attacks in Paris killed 130 people, the European Union began pressuring tech companies to work harder to prevent the spread of violent extremism online.
After a year of negotiations, Facebook, Microsoft, Twitter and YouTube agreed to the European Union's hate speech code of conduct, which commits them to review and remove the majority of valid complaints about illegal content within 24 hours and to be audited by European regulators.
The first audit, in December, found that the companies were only reviewing 40 percent of hate speech within 24 hours, and only removing 28 percent of it. Since then, the tech companies have shortened their response times to reports of hate speech and increased the amount of content they are deleting, prompting criticism from free-speech advocates that too much is being censored.
Now the German government is considering legislation that would allow social networks such as Facebook to be fined up to 50 million euros if they don't remove hate speech and fake news quickly enough.
Facebook recently posted an article assuring German lawmakers that it is deleting about 15,000 hate speech posts a month. Worldwide, over the last two months, Facebook deleted about 66,000 hate speech posts per week, vice president Richard Allan said in a statement Tuesday on the company's site.
Among posts that Facebook didn't delete were Donald Trump's comments on Muslims. Days after the Paris attacks, Trump, then running for president, posted on Facebook "calling for a total and complete shutdown of Muslims entering the United States until our country's representatives can figure out what is going on."
Candidate Trump's posting — which has come back to haunt him in court decisions voiding his proposed travel ban — appeared to violate Facebook's rules against "calls for exclusion" of a protected religious group. Zuckerberg decided to allow it because it was part of the political discourse, according to people familiar with the situation.
However, one person close to Facebook's decision-making said Trump may also have benefited from the exception for sub-groups. A Muslim ban could be interpreted as being directed against a sub-group, Muslim immigrants, and thus might not qualify as hate speech against a protected category.
Hannes Grassegger is a reporter for Das Magazin and Reportagen Magazine based in Zurich.
ProPublica is a Pulitzer Prize-winning investigative newsroom. Sign up for their newsletter.