wow, discord acknowledged on the Bad Place that it was true that an overfitted CSAM detection AI started flagging grids (spreadsheets, game inventory screenshots, etc) as illegal and instabanning people.

8200 people got banned this way over several weeks before someone managed to get their attention on reddit. But the good news is they have in fact manually rolled back all the bans.

On the one hand, this is the one problem where you absolutely want as little hands-on human moderation as possible. On the other, this is a very clear-cut overfitting failure mode on a small dataset (of the images that governments provide to major websites for automated detection) with disastrous consequences for random innocent people. And they presumably had to go back and manually check all bans in this category for the last few weeks, thus maximizing the moderators' exposure too. 🤦‍♀️

@0xabad1dea I've heard that putting grids over illegal content was a common practice to avoid automated filtering. With this incident in mind bad actors can move on to deliberate dataset poisoning (assuming the raising FP rate was accidental this time) :(

@0xabad1dea From what an employee said, it wasn't AI. It was PhotoDNA (the ages old CSAM perceptual image hashing thing). My guess is NCMEC gave them some bad hashes that match random stuff like grids. PhotoDNA works on a 6x6 grid, so it makes sense that uniform hash data patterns would match certain gridlike images.

The bigger screwup is allowing those matches, especially for newly added hashes, to ban people without manual review first, and then the thing where apparently their manual unban was itself broken.

@0xabad1dea Since it was bogus hashes, they know exactly who was banned for the bad ones without manually reviewing everything. It's not like an AI that just outputs a prob, it's a match against a specific hash.

@0xabad1dea ok, but someone posting a screenshot of Excel SHOULD be banned