New York City’s business chatbot gives advice that would break the law
An investigation found that New York City’s official chatbot for business owners gave answers that contradicted city and state law. The city kept it online with disclaimers until 2026.
- Disclosed
- Published
- Updated
- Organization
- City of New York
- System
- MyCity government chatbot built on Microsoft Azure AI services
- Impact
- Wrong legal guidance on tips, cash payments and housing vouchers given to business owners; chatbot taken down in February 2026
Timeline
- October 2023. The city announces the chatbot as a pilot.
- 2024-03-29. The Markup publishes its findings.
- April 2024. The mayor says the chatbot is wrong in some areas and will be fixed. The city adds disclaimers.
- 2026-01-30. The new mayor calls the chatbot “functionally unusable” and plans to take it down.
- 2026-02-04. The Markup reports that the city has taken the chatbot down.
Trigger or entry point
Ordinary questions from business owners about city rules.
Failure mechanism
The reports give examples of wrong answers and do not explain why the model produced them.
Impact
According to The Markup, the chatbot said that a restaurant could go cash-free, that employers could take a cut of workers’ tips, and that landlords did not have to accept housing vouchers. Each answer contradicts the law, The Markup reports. No case of a business being penalised for following the advice is reported.
Response
In 2024 the city kept the chatbot online. Reuters quotes the mayor as saying, “It’s wrong in some areas, and we’ve got to fix it.” The city added a notice that responses are not legal or professional advice. In January 2026 the new administration said the chatbot cost about half a million dollars and announced its removal.
Mitigations
Disclaimers in 2024, then removal of the chatbot in February 2026.
Sources
- NYC’s AI Chatbot Tells Businesses to Break the Lawsecondary · 2024-03-29
- New York City defends AI chatbot that advised entrepreneurs to break laws (Reuters)secondary · 2024-04-05
- Mamdani to kill the NYC AI chatbot we caught telling businesses to break the lawsecondary · 2026-01-30
Corrections
No corrections have been made to this entry.
Report a correction