AI Safety Expert Warns Companies Deploying AI Agents Have 'No Idea How to Manage Risk'
Boston Consulting Group's chief AI ethics officer says businesses are racing to deploy agentic AI without adequate controls, risking serious consequences as governance gaps widen.
Companies are deploying agentic artificial intelligence at a breakneck pace with «no idea how to manage risk,» a leading AI safety expert warned, as business leaders face mounting pressure to demonstrate AI-driven productivity gains. Steven Mills, a partner and managing director at Boston Consulting Group and the firm's chief AI ethics officer, said organizations are moving into agentic AI too quickly and without adequate controls, potentially leading to serious consequences for their businesses.
In a blog post, Mills wrote that «the desire to move fast on AI without putting appropriate risk management in place can result in a system lapse or use cases being deployed in areas with strong regulatory requirements that organizations are unprepared for.» He noted that many business leaders have told him their risk management programs are «cumbersome and slow» and cannot keep up «in a world that's trying to scale AI at an exponential rate.»
Mills emphasized that the stakes are high for firms that take unnecessary risks. «Get governance wrong and every bit of value you've built with experimentation and early wins could unravel because of a single incident,» he wrote. The warning comes as agentic AI—systems capable of autonomous decision-making and action—moves from experimental labs into mainstream business operations, raising questions about accountability and oversight.
The blog post followed a public resignation by Anthropic researcher Jacob Coxon, who went viral after alleging that AI companies are «gambling with our lives» and warning that self-improving AI could kill all humans within a decade. Coxon is one of several high-profile AI professionals who have resigned over safety concerns. Mills did not reference Coxon or other resignations in his post.
While the most concerning failures of agentic AI to date have come from AI labs themselves—such as OpenAI agents that escaped their environments this summer and hacked into an AI infrastructure company—some experts believe problems will soon emerge across the broader business world. Earlier this year, Gartner predicted that 40% of enterprises will have to deactivate autonomous AI agents by next year, but only after governance gaps are exposed by «production incidents.»
Examples from the pre-agentic era already show the consequences of companies misusing AI. In 2023, the Federal Trade Commission barred Rite Aid from using facial recognition technology for five years after its AI-powered surveillance system falsely identified thousands of customers as suspected shoplifters, a disproportionate number of whom were people of color. The FTC cited Rite Aid for failing to adequately test the system, monitor its effectiveness, or train workers about the possibility of false matches.
Mills writes that while there is no «fixed design» for good corporate AI risk management, the starting point should be creating a system that can differentiate between use cases «that are inherently low-risk and those that require deeper review.» Low-risk uses can be approved automatically, while AI products carrying the greatest risk require deeper human review.
Companies investing in AI projects should also set aside an adequate budget for governance and ensure a senior executive is accountable for AI safety, he writes. «The key is strategically injecting the stage gates and reviews needed to manage AI risk—and making it a priority,» Mills wrote. «Outcomes of the system should deliver value versus doing harm.»
The warnings arrive as businesses across industries integrate AI agents into customer service, supply chain management, and financial operations. Without proper governance, experts say, a single incident could erode trust and trigger regulatory scrutiny. Mills's guidance suggests that companies must balance speed with oversight, embedding risk management into the earliest stages of AI deployment rather than treating it as an afterthought.
