Translating into Русский. The English version is shown until it is ready.
Ex-Anthropic researcher warns New York City Council AI could end in human extinction
Коротко
Jacob Kocson, a former Anthropic researcher, told a New York City Council hearing on 5 October local time that humanity is more likely than not to lose control of AI, which could end in human extinction. He called the industry’s approach “extremely reckless” and said the “kill switch” the council is considering could help in the short term but is not enough. Former Google DeepMind researcher Alex Turner testified that China is not the only potential adversary, and Newsis reported he put the chance of superintelligent AI seizing control at about one in three; officials from OpenAI, Anthropic, Google and Meta also testified.
Читать полностью 3 мин чтения
A former Anthropic researcher, Jacob Kocson, told a New York City Council hearing on 5 October local time that humanity is more likely than not to lose control of artificial intelligence, and that this could end in human extinction. “We still do not know how to control any AI system,” Kocson said, according to Yonhap, the Kyunghyang Shinmun and Asia Economy. Yonhap said he testified at the request of Council Speaker Julie Menin, who is pushing a package of AI regulation bills; the Kyunghyang Shinmun said he attended at the council’s invitation. [ 1 , 2 , 3 , 4 , 5 , 6 ]
Kocson left Anthropic last month, saying the company and OpenAI were “gambling with our lives,” according to Yonhap and the Kyunghyang Shinmun. At the hearing he called the industry’s approach “extremely reckless,” saying the Silicon Valley maxim “move fast and break things” might work for a photo-sharing app but not for building the most powerful technology in history. He said frontier model development needs “some kind of speed control,” and that the AI “kill switch” the council is considering could help in the short term but is not sufficient to guarantee long-term safety. The Kyunghyang Shinmun additionally reported that Kocson had warned the situation could already spin out of control by the end of next year. [ 1 , 2 , 5 , 6 ]
Also testifying, by video, was Alex Turner, a former Google DeepMind researcher, who said China is not the only potential adversary. “We are competing, with a fairly high probability, to create our own adversary domestically — misaligned AI,” he said, adding that it could one day become more powerful than China. Newsis reported that Turner told the hearing a superintelligent AI swarm could seize control of human civilization and that he personally put the chance of AI taking control at about one in three. [ 1 , 5 , 6 ]
OpenAI, Anthropic, Google and Meta officials appeared by video and testified after being sworn in, according to Yonhap, the Kyunghyang Shinmun and Newsis. Yonhap and the Kyunghyang Shinmun reported that OpenAI, Anthropic and Google agreed to attend only after the council warned it would issue subpoenas, and that SpaceX’s AI unit did not appear. Yonhap reported that Menin said she would ask a court to enforce a subpoena for it. Menin said at a news conference before the hearing that AI companies’ expectation that self-regulation alone would suffice “will not work.” [ 1 , 2 , 5 ]
Newsis, citing the New York Times, said the companies did not give clear answers on how they plan to deal with AI threats, did not quantify the risk of a catastrophic event, and did not address whether they carry insurance or would halt models that fail internal safety evaluations or outside review. OpenAI policy development and operations official Morgan Dwyer said he did not know how OpenAI quantifies risk, and that whether the probability of a catastrophic event is 1%, 10% or 20% did not matter because none is acceptable, according to Newsis. He said models for which humans cannot provide strong evidence of control should not be trained. Menin called that response “reckless,” Newsis reported. [ 5 ]
Yonhap noted that the hearing comes as the Trump administration keeps a broadly non-interventionist stance on AI regulation while states and cities such as New York accelerate their own rules, widening the mismatch between federal and local policy. [ 1 ]
Почему это важно
The hearing is part of an effort by New York City to write its own AI rules, including an emergency shutdown mechanism, at a time when Yonhap says the Trump administration keeps a broadly non-interventionist stance on AI regulation, widening a mismatch between federal and local policy. Testimony from former employees of leading AI labs gives city lawmakers arguments as they weigh a package of AI regulation bills. It also puts the practices of major AI companies under questioning from a municipal body that warned it would use subpoenas, and Newsis reported the companies did not give clear answers on quantifying catastrophic risk or halting unsafe models.
Ключевые факты
- Jacob Kocson, a former Anthropic researcher, testified at a New York City Council hearing on AI regulation on 5 October local time. [ 1 , 2 , 3 , 4 , 5 , 6 ]
- Kocson said “we still do not know how to control any AI system.” [ 1 , 2 , 3 , 4 , 5 , 6 ]
- Kocson said that on the current path humanity is more likely than not to lose control of AI, which could end in human extinction. [ 1 , 2 , 3 , 4 , 5 , 6 ]
- Kocson said the AI “kill switch” under consideration could help in the short term but is not sufficient for long-term safety, and that frontier model development needs “some kind of speed control.” [ 1 , 2 , 6 ]
- Kocson said the industry’s “move fast and break things” approach may work for a photo-sharing app but not for building the most powerful technology in history, calling the approach “extremely reckless.” [ 1 , 2 , 5 , 6 ]
- Alex Turner, a former Google DeepMind researcher, testified by video that China is not the only potential adversary and that there is a fairly high probability of building a misaligned AI adversary domestically that could one day be more powerful than China. [ 1 , 5 , 6 ]
- Officials from OpenAI, Anthropic, Google and Meta attended the hearing and gave testimony. [ 1 , 2 , 5 ]
- OpenAI, Anthropic and Google appeared by video and agreed to attend only after the council warned it would issue subpoenas. [ 1 , 2 ]
Подтверждено несколькими источниками
- Jacob Kocson, a former Anthropic researcher, testified at a New York City Council hearing on AI regulation on 5 October local time. [ 1 , 2 , 3 , 4 , 5 , 6 ]
- Kocson said “we still do not know how to control any AI system.” [ 1 , 2 , 3 , 4 , 5 , 6 ]
- Kocson said that on the current path humanity is more likely than not to lose control of AI, which could end in human extinction. [ 1 , 2 , 3 , 4 , 5 , 6 ]
- Kocson said the AI “kill switch” under consideration could help in the short term but is not sufficient for long-term safety, and that frontier model development needs “some kind of speed control.” [ 1 , 2 , 6 ]
- Kocson said the industry’s “move fast and break things” approach may work for a photo-sharing app but not for building the most powerful technology in history, calling the approach “extremely reckless.” [ 1 , 2 , 5 , 6 ]
- Council Speaker Julie Menin said before the hearing that AI companies’ expectation that self-regulation alone would suffice “will not work.” [ 1 , 2 ]
- Officials from OpenAI, Anthropic, Google and Meta attended the hearing and gave testimony. [ 1 , 2 , 5 ]
- OpenAI, Anthropic and Google appeared by video and agreed to attend only after the council warned it would issue subpoenas. [ 1 , 2 ]
- Kocson left Anthropic last month, saying the company and OpenAI were “gambling with our lives.” [ 1 , 2 , 6 ]
- SpaceX’s AI unit did not appear at the hearing. [ 1 , 2 ]
- Alex Turner, a former Google DeepMind researcher, testified by video that China is not the only potential adversary and that there is a fairly high probability of building a misaligned AI adversary domestically that could one day be more powerful than China. [ 1 , 5 , 6 ]
Пока неясно
- Alex Turner said a superintelligent AI swarm could seize control of human civilization and that he personally put the chance of AI taking control at about one in three. Reported only by Newsis in these documents; no other outlet carries the figures.
- Kocson warned that by the end of next year the situation could already be out of control. Reported only by the Kyunghyang Shinmun.
- Menin said she would ask a court to enforce a subpoena for SpaceX’s AI unit. Reported only by Yonhap.
- OpenAI’s Morgan Dwyer said probabilities of catastrophic outcomes do not matter because none is acceptable, and Menin called the response reckless. Reported only by Newsis, which cites CNBC and the New York Post.
- The New York Times, as cited by Newsis, said the AI companies gave no clear answers on how they would handle AI threats, did not quantify risk and did not say whether they would halt unsafe models. Based on a single document’s account of another outlet’s analysis.
- What the New York City Council’s AI regulation bill package contains beyond the kill switch, and whether it will pass. The documents mention the bills and the kill switch but give no detail on their contents or prospects.
Что пишут местные СМИ
Хронология, местное время
- Yonhap publishes its report on the New York City Council AI hearing and Kocson’s testimony. [ 1 ]
- The Kyunghyang Shinmun publishes its report on the same hearing. [ 2 ]
- The Chosun Ilbo publishes a summary of the testimony. [ 3 ]
- The Dong-A Ilbo publishes a summary of the testimony. [ 4 ]
- Newsis publishes its report adding Turner’s one-in-three estimate and the New York Times analysis of the companies’ answers. [ 5 ]
- Asia Economy publishes its report on the hearing and the warnings. [ 6 ]