VMTech
Discuss a project →

OpenAI adds CSAM classifier and ChatGPT safety warnings

OpenAI adds CSAM classifier and ChatGPT safety warnings

OpenAI has announced a new image classifier for detecting child sexual abuse material (CSAM), alongside ChatGPT warnings and temporary account blocks for attempts to create or access content that sexualizes children. In early testing with partners, the classifier flagged 48% fewer images while finding the same number of abusive images, and the images it flagged were twice as likely to be abusive.

The company is offering selected companies no-cost access to the classifier through its API. OpenAI developed the tool with the National Center for Missing and Exploited Children (NCMEC) and is working with Tech Coalition and ROOST to share access.

Classifier is designed to support reviewers

The classifier examines images for visual indicators of abuse, including images generated by AI that it has not encountered before. It assigns a score intended to give reviewers an indicative direction when handling potential CSAM. OpenAI’s reported testing result points to fewer unnecessary flags without reducing the number of abusive images found.

The tool is intended to sit within a broader operational response rather than replace judgment by trained reviewers. Its release follows sustained scrutiny of AI providers over child safety, while non-consensual sexualized imagery has also been created through other AI chatbots.

ChatGPT deterrence and developer guidance

ChatGPT already blocks CSAM and related requests. The new warning screens will appear when a user attempts prompts seeking material that sexualizes children, directing the user to specialised support intended to stop harmful behaviour and applying a 24-hour account suspension as a warning. OpenAI said repeat offences may result in permanent account removal.

OpenAI formed its messages and guidelines with Protect Children, the Lucy Faithfull Foundation and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of deterrent messages. The company also published guidance for developers using its tools to prevent, detect and help users report CSAM. In early testing, it said the guidance helped developers adopt child-safety tooling and register to report CSAM to NCMEC.

Support for investigators

For law-enforcement teams working on child-safety cases, OpenAI is providing AI-powered capabilities including dedicated plugins, investigative tools and complex data analysis. Access will be managed by programme administrators, while investigators will receive training and a field guide for using AI.

The announcement comes as OpenAI continues to adjust its safety controls. Concerns around governance are also sharpened by OpenAI safety staff data-sharing dispute as the company expands systems that depend on rigorous safeguards, clear reporting channels and accountable operational processes.

For businesses deploying generative AI, the practical implication is to build child-safety controls into product workflows: pair automated detection with human review, define escalation and reporting procedures, and make repeat violations subject to clear account enforcement.

#openai#aisafety#childsafety#contentmoderation
Open analytics
On the site 2 views
min read 3 16.11.2026
Instagram

OpenAI adds CSAM classifier and ChatGPT safety warnings

Open the post on Instagram ↗