VMTech
Discuss a project →

OpenAI expands child-safety tools with CSAM detection and support

OpenAI expands child-safety tools with CSAM detection and support

OpenAI has released a new image classifier designed to detect potential child sexual abuse material (CSAM), including a dedicated signal indicating whether material may be AI-generated. The company is offering the classifier free through the OpenAI API to vetted organizations, alongside new ChatGPT interventions, developer guidance and support for child-safety law-enforcement units.

In an evaluation of user-uploaded images, OpenAI estimated that the classifier would flag 48% fewer images than its existing protection while finding roughly the same number of abusive images. Images flagged by the new system were about twice as likely to contain abuse, the company said. The classifier returns scores intended to help human reviewers prioritize images requiring attention.

A classifier aimed at novel material

OpenAI said the classifier examines visual signs of abuse and can help identify images not previously encountered. It described the release as the first broadly available CSAM classifier with a specific signal for whether CSAM may be AI-generated. The system was trained, evaluated and designed for deployment with assistance from the National Center for Missing and Exploited Children (NCMEC), which operates the CyberTipline reporting hub.

Early partners supplied further results from their own evaluations. fal reported that human reviewers confirmed 72% of high-confidence flags as novel CSAM in production-image testing. invideo AI said one in four flagged cases resulted in a ban. In SafetyKit evaluations involving drawings or animations that sexualize children, the classifier detected 90% of cases.

OpenAI plans to distribute access with help from the Tech Coalition and ROOST, and said it supports other tools, including Thorn’s Safer CSAM detection system. It has also joined the Tech Coalition’s Lantern program to share and receive signals related to online child sexual exploitation and abuse across platforms. The company said feedback from internal testing and early partners helped identify weaknesses and improve the classifier.

Warnings, guidance and investigative support

Within ChatGPT, OpenAI is adding product warnings to interrupt attempts to locate abusive material. The warning states that sexualizing children causes harm, including where requested content is AI-generated. OpenAI said it will remove the flagged conversation and pause ChatGPT access for 24 hours while making specialist support available to help stop harmful behaviour.

Repeated violations may result in longer restrictions, account closure or reports to authorities. OpenAI developed the intervention with input from Protect Children, the Lucy Faithfull Foundation, Dr. Caoilte Ó Ciardha of the University of Kent, and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of deterrent messages.

The company has also published CSAM guidance for developers, created with input from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation. It covers rules, user reporting, detection, review and reporting processes, safety identifiers and support for teams exposed to sensitive material. OpenAI said developers can combine that guidance with its classifier, the Moderation API, ROOST’s Coop review console and dedicated systems such as Thorn’s Safer.

Tools for specialist investigators

OpenAI is also preparing tools and training for child-safety law-enforcement units. At launch, its first investigative tool is intended to help authorized teams review long chat logs and legal-process returns, identify potential connections and build timelines with findings linked back to source material. Access will be managed by program administrators and will depend on agency authorization and program requirements.

For businesses building or operating AI services, the announcement underscores that child-safety controls need more than a detection model: they require human review, user reporting paths, escalation procedures, support for reviewers and cooperation with specialist organizations and investigators.

#childsafety#aisecurity#contentmoderation#trustandsafety
Open analytics
On the site 1 views
min read 4 16.11.2026
Instagram

OpenAI expands child-safety tools with CSAM detection and support

Open the post on Instagram ↗