Flock Safety's AI Police Search Tool Flags Abuse Risks Its Guardrails Don't Stop
The AMW Read
Independent technical analysis shows Flock's AI moderation classifier flags but never blocks abusive police searches, sharpening the known safety/backlash narrative with a concrete mechanism and tying it to Illinois's state-law data-access finding.
Flock Safety's AI Police Search Tool Flags Abuse Risks Its Guardrails Don't Stop
WIRED reverse-engineered the code behind Flock Safety's latest police search suite, which now lets officers find people on camera by written description, not just by license plate. The toolset includes an AI watchlist that runs a continuous automated search for a description across every camera in a cordoned-off area, a plain-language person search called FreeForm, and a Smart Sort feature that reranks footage as officers approve or reject results. Flock added guardrails in August — shorter default data retention, mandatory case codes, automated audit logging, mandatory by year-end — after a Texas deputy searched more than 83,000 cameras for a woman who had an abortion, two members of Congress wrote CEO Garrett Langley, and Illinois found the company let federal immigration agents reach state camera data in violation of state law. WIRED's analysis found Flock's model scores each search query against sensitive categories including race, religion, and political or cultural expression, but only political/cultural expression triggers no hard stop, and none of the categories actually block a search — they log it and let the officer proceed.
The deeper story is architectural: a moderation classifier scores officer queries against sensitive categories, but the decision logic runs on Flock's servers, is invisible to the departments deploying it, and can't be independently audited for accuracy or bias. Flock Safety is tracked in the AI Market Watch index with $1.16B in total funding raised since its 2017 founding, per the AI Market Watch index (which tracks roughly 5,000 companies — coverage, not a census), underscoring how much capital has flowed into a company whose central safety decision sits inside an unaudited classifier rather than a transparent, enforceable policy.
For builders selling classification or moderation layers into regulated, high-stakes deployments — policing, hiring, lending — the lesson is that a scoring model paired with a soft warning isn't a control; it records a decision without preventing it, and that gap is what regulators and plaintiffs' lawyers will target next. Investors evaluating public-safety AI vendors should treat unaudited moderation architecture as a liability line item, especially as more states follow Illinois in scrutinizing cross-agency and cross-jurisdiction data access.
#FlockSafety #PoliceAI #Surveillance #AIGovernance #PublicSafety #ContentModeration