OpenAI opens a free detector for child abuse images to other companies
The tool also recognises AI-generated images; anyone who tries to request such content in ChatGPT will now be suspended for a day.
OpenAI has released an image classifier designed to detect child sexual abuse material, including AI-generated images that have not been catalogued anywhere. According to TechCrunch, the system gives each image a score that helps moderators decide how to handle it. It was developed with the US National Center for Missing and Exploited Children (NCMEC). Selected companies get free access through the API, and OpenAI is working with the Tech Coalition and ROOST to widen that access.
In tests with partners the classifier flagged 48 percent fewer images while still finding the same amount of abuse material, TechCrunch reports, so flagged images were twice as likely to actually contain abuse. That saves work for the people who have to review them. OpenAI's developer documentation shows that organisations must apply, complete an identity check through Persona and accept terms. The same documentation notes that OpenAI's standard moderation model only assesses text in that category, not images.
Anyone who tries to obtain content sexualising children in ChatGPT will now get a warning, a referral to specialised support and a 24-hour suspension; repeat offences can lead to permanent removal of the account. The messages were drawn up with Protect Children, the Lucy Faithfull Foundation and researcher Joel Scanlan of the University of Tasmania. TechCrunch notes that OpenAI faces several lawsuits over harm to users, especially children.