Content moderation: take a post down, or send it to a moderator¶
A trust-and-safety classifier has flagged a post, and the platform must take it down or leave it up. Most violating content is not new: it is a re-upload of something a moderator already ruled on, slightly cropped, re-encoded or captioned. This page shows how the library takes a post down automatically only when it clearly belongs to a tight cluster of copies that moderators ruled on consistently, and sends it to a moderator when the cluster might just be many different posts sharing a template. It is written for a trust-and-safety policy lead or moderation-operations manager; the maths is in collapsed boxes and linked pages.
At a glance
| Decision | Take the post down (take_down) or leave it up (leave_up). |
| Owner | Trust-and-safety policy; moderation operations handles holds. |
| Today's rule | Classifier flags it (flagged): take down. Otherwise (benign): leave up. |
| What this adds | Before acting, check that the post's near-duplicate cluster is clearly copies of one piece of content and that moderators ruled on those copies consistently. Otherwise hold for a moderator. |
| Demo outcome | 3 posts: 2 acted on, 1 held (a post that resembles a loose cluster of template posts). |
| Example id | content_moderation (uv run grounded-decision examples content_moderation) |
The decision today¶
| Mistake | What it costs |
|---|---|
| Take down a legitimate post | A silenced user, an appeal, a statement of reasons you must defend, lost trust. |
| Leave up a violation | The content spreads; for some classes, legal exposure. |
| Hold for a moderator | Moderator time, and the post stays in its interim state until reviewed (see What happens next). |
Where today's rule goes wrong¶
Take three posts:
- Post 5501 is a re-upload of a scam video moderators have already removed 20 times. Its perceptual hash (a fingerprint that survives cropping and re-encoding) is within a few bits of all 20 earlier copies, and every copy is within a few bits of every other: one piece of content. Earlier rulings transfer. Take it down.
- Post 5502 is also flagged as a scam video, and it resembles 30 previously reviewed posts. But those 30 are only loosely alike: each matches about a third of the others. That is what a popular meme template looks like: many different posts with a shared layout, whose rulings depended on what each one said. Whether this cluster is "one piece of content" or "many posts" is a coin flip.
- Post 5503 is a copy of a news photo that moderators allowed 20 times. Leave it up.
Today's rule takes down 5502 as confidently as 5501. Taking it down treats a coin flip as a ruling.
What the system checks, in your terms¶
The library sees the classifier's call as an event (flagged or benign),
plus a summary of the post's related cases: previously reviewed posts
whose perceptual hash is close to this one, a link between two of them when
their hashes differ in at most 10 of 64 bits, and each post's ruling. All
six checks must pass for the proposed action to run; two check the policy
itself and pass on every row (see the six checks).
Three of the four that read your data are below; the fourth, support, is not measured in this example and passes with support not measured.
| Check | Question, in moderation terms | Reads | Fails when | What a hold means |
|---|---|---|---|---|
| Neighbour agreement | Are there enough near-duplicates, ruled on consistently enough, that earlier rulings say something about this post? | Near-duplicate links and each post's ruling | Few matches, or matched posts were ruled inconsistently | "Earlier rulings on similar posts don't agree enough to transfer." |
| Knife-edge | Is the cluster clearly one piece of content (or clearly many posts), rather than on the line where a few extra matches would flip that? | Cluster size, number of links, your "one piece of content" cut-off (0.30 link density) | Link density is inside the uncertainty window around the cut-off | "Can't tell whether this is one piece of content or a template." |
| Unusual cases | Does the classifier's score for this post look like its scores on recent decisions that were upheld on appeal or audit? | Classifier unusualness score; the last 30 upheld decisions' scores (the calibration set) | This post's score is more extreme than almost all of them | "The classifier hasn't been right on posts like this recently." |
What these checks do NOT verify
Direction. The agreement check asks whether matched posts were
ruled consistently with each other, not whether those rulings
support this action. Reproduced with the shipped
content_moderation functions: a benign event on post 5501's
cluster (20 copies, all removed) returns act leave_up. The
direction comes from the classifier; the library only asks whether
the cluster is informative enough to act without a moderator. The
support check would hold that
leave_up if the evidence carried the share of matched posts left up
(here 0 of 20); this example does not measure it.
Which side of the line. The knife-edge check holds only when
density is near the cut-off. A very loose cluster, clearly "many
different posts" whose rulings should not transfer, is far from
the cut-off and passes. Reproduced: 30 posts, all ruled violating,
with link density 0.083 (well below 0.30) acts take_down. If loose
clusters should never transfer rulings in your policy, send a
different event for them (for example, flagged_novel, which
proposes a review rather than a take-down) based on the density you
compute upstream.
Data you provide¶
| Input | Source system | Refresh | Owner | Minimum / quality bar |
|---|---|---|---|---|
Reviewed posts near this one in hash space (ReviewedPost: post id, 64-bit perceptual hash, ruling) |
Near-duplicate index + moderation decisions table | Per flagged post | T&S engineering | Rulings that were overturned on appeal should carry the final ruling. Never include the post being decided. |
Near-duplicate distance (NEAR_DUPLICATE_MAX_HAMMING_DISTANCE, 10 bits) |
Tuned on pairs moderators labelled "same content" / "different" | Quarterly | T&S data science | Check it per content type; images and video behave differently. |
One-piece-of-content cut-off (SINGLE_CONTENT_DENSITY_THRESHOLD, 0.30) |
Your history | Quarterly | T&S data science | The density at which past clusters were most often split by reviewers. See Choose thresholds and calibration. |
| Classifier unusualness score for this post | Classifier | Per post | T&S data science | Same scale as the calibration set. |
Calibration set: scores of the last 30 decisions upheld on appeal or audit (RECENT_UPHELD_DECISION_SCORES) |
Appeals and audit outcomes | Daily | Moderation ops | At least 9 at the default 10 % level. Appeals are filed mostly on take-downs, so this set skews toward them. |
near_duplicate_evidence in content_moderation.py builds the evidence
summary from the reviewed posts.
Worked example¶
These rows come from uv run grounded-decision examples content_moderation.
| Post | Event | Cluster | Agreement check | Knife-edge check | Unusual-cases check | Today's rule | This system |
|---|---|---|---|---|---|---|---|
| 5501 | flagged (scam video) |
20 copies, all 190 pairs linked, 20 removed | Pass | Pass: clearly one piece of content | Pass | take down | act take_down |
| 5502 | flagged (scam video) |
30 posts, 161 links, 30 removed | Pass: rulings agree | Fail: density right on the one-content line | Pass | take down | hold; proposed take_down kept on the record |
| 5503 | benign |
20 copies, all linked, 0 removed | Pass | Pass | Pass | leave up | act leave_up |
Note that row 5502 is held by the knife-edge check alone: the 30 template
posts were all ruled violating, so their rulings agree perfectly. The
question is whether this post is one of them, and the cluster's shape
can't say. Over three posts, today's rule acts three times; this system
acts twice and holds once (blocked_overconfident_acts: 1 in the --json
summary).
The numbers behind each check
\(d\) is the average number of links per post, \(\lambda\) how much more often linked posts share a ruling than chance (0 = chance, 1 = always). Evidence strength \(d\lambda^2\) must exceed 1.05.
| Post 5501 | Post 5502 | Post 5503 | |
|---|---|---|---|
| posts \(n\) | 20 | 30 | 20 |
| links per post \(d\) | 19 | 10.73 | 19 |
| agreement \(\lambda\) | 1 | 1 | 1 |
| evidence strength \(d\lambda^2\) | 19 (pass) | 10.73 (pass) | 19 (pass) |
| link density \(\hat p\) | 1.0 | 0.370 | 1.0 |
| knife-edge half-width at this \(n\) | 0.179 | 0.139 | 0.179 |
| distance from the 0.30 cut-off | 0.70 (pass) | 0.070 (inside: fail) | 0.70 (pass) |
| unusualness p-value | 1.0 | 1.0 | 1.0 |
Half-width is \(\ln(1/(2\varepsilon))/(\ln n)^2\) with
\(\varepsilon = 0.1\), capped at 0.5 (computed with uv run python -c).
The demo sends the same classifier score, 0.10 (TYPICAL_POST_SCORE),
for every post, below every calibration score (0.35 to 0.93), so
the unusual-cases check cannot fail in this demo.
The record's reason for post 5502 is
evidence density inside operational window (theorem width=7.29e+04, vacuous=True):
"operational window" is the knife-edge window, and the proven width
is reported but is far wider than 1, so it is not used.
What happens next¶
| Verdict | Who is told | What they see | Time limit | If nobody acts in time | Feedback |
|---|---|---|---|---|---|
Act take_down |
Poster (notice with statement of reasons), moderation dashboard | action=take_down, the decision record |
Immediate | n/a | Appeals and their outcomes feed the calibration set and the post's ruling. |
Act leave_up |
Nobody | action=leave_up |
Immediate | n/a | Later reports re-enter the flow. |
| Hold | Moderation queue | proposed_action=take_down, failed checks, the near-duplicate cluster |
Set by content class and severity: for example, hours for a suspected scam. | Apply your interim state (for example, limit distribution or reduce reach) and escalate; do not convert the hold into a take-down automatically. | The moderator's ruling is added to the cluster, so the next copy has a clearer picture. |
Content where holding is not acceptable. For child sexual abuse material, terrorist content subject to removal orders, and other classes with legal removal deadlines, a hold that leaves the post up is not a lawful option. Route those classes around this module (hash-match against the mandated databases and remove), or define the hold for them as "remove pending review" in your own code. The library has no notion of content class.
Notices and appeals. The decision record says which checks passed and which failed. It is not a statement of reasons: a user notice must name the rule broken and the facts relied on, which come from your policy and the moderator, not from the check names.
Tuning and expected workload¶
| Setting | Policy or data? | How to choose | Too tight | Too loose |
|---|---|---|---|---|
| Near-duplicate distance (10 bits) | Data | Pairs moderators labelled same/different | Copies not linked; clusters look loose | Different posts linked; templates look like one content |
| One-content cut-off (0.30) | Data | Density at which reviewers most often split past clusters | — | — (both directions move holds, see knife-edge below) |
| Agreement margin (0.05) | Policy | Keep the default | More holds | Inconsistent rulings transfer |
| Unusualness level α (0.10) | Policy | Share of upheld decisions you accept treating as unusual | More holds | Odd scores act automatically |
Expected holds. Holds come from clusters whose density lands within the window of the cut-off. The window's half-width is 0.18 at 20 posts, 0.14 at 30, 0.084 at 80 and 0.057 at 200. With a 0.30 cut-off and clusters of about 30, anything with density between about 0.16 and 0.44 is held. Measure the share of your flagged posts in that band before launch: Estimate impact before launch.
Limits, risks and compliance¶
- Direction is not checked, and very loose clusters pass (see the warning above).
- The demo's unusual-cases check cannot fail, because it sends a fixed score.
- Synthetic data. The clusters are generated by flipping hash bits at random (3 % for the tight clusters, 10 % for the template), not drawn from a real index.
- Context changes meaning. The same image can be news reporting in one post and glorification in another. Hash clusters cannot see captions or intent.
- Calibration skew. Appeals come mostly from take-downs, so the calibration set says little about posts that were left up.
Wiring it in¶
- At service start, build the policy once:
moderation_module = build().flaggedproposestake_down,benignproposesleave_up. - Daily, refresh
RECENT_UPHELD_DECISION_SCORESfrom appeal and audit outcomes. - Per flagged post, look up reviewed posts near it in the hash index and build the evidence:
from grounded_decision.examples.content_moderation import near_duplicate_evidence
evidence = near_duplicate_evidence(
reviewed_posts, # [ReviewedPost(post_id, perceptual_hash, ruling), ...]
candidate_classifier_score=classifier_score_for_this_post,
)
record = moderation_module.decide(event, evidence)
append_to_audit_log(record.to_dict())
if record.verdict.value == "act":
execute(record.action)
else:
apply_interim_state(post_id) # e.g. limit distribution
enqueue_for_moderator(post_id, record.proposed_action, record.reason)
append_to_audit_log, execute, apply_interim_state and
enqueue_for_moderator are your own functions.
- Concurrency. The module is not thread-safe; see Build a decision service.
Run it yourself¶
uv run grounded-decision examples content_moderation
uv run grounded-decision examples content_moderation --json
How to read the printed report:
Example library: reading a report. HOLD
and hold_for_human in the output are this page's "hold" (the record's
verdict field says abstain).
Further reading¶
- Naive vs grounded: the same pattern across cases.
- Live: trust-and-safety take-down queue (
live_trust_queue). - Build evidence from your data and Troubleshoot: why did it abstain?.
- Theory: knife-edge densities (Friedgut–Kalai), recovering a label from related cases (Kesten–Stigum), conformal prediction.
Results this case relies on
| Check | Result | Math catalog number | Trust level of the check |
|---|---|---|---|
| Knife-edge | Friedgut–Kalai sharp thresholds | 186 | practical (operational): the proven width (about \(7 \times 10^4\) at 30 posts) is far wider than 1 |
| Neighbour agreement | Kesten–Stigum reconstruction | 229 | proven (lean) above the line |
| Unusual cases | Split conformal prediction | — | published |
| Policy shape and progress | Star height; parity and mean-payoff games | 134, 104 | proven (lean) |