May 16th 2018

Preventing an AI Apocalypse

by Seth Baum

Seth Baum is the executive director of the Global Catastrophic Risk Institute (GCRI), a think tank focused on extreme global risks. He is also affiliated with the Blue Marble Space Institute of Science and the University of Cambridge Centre for the Study of Existential Risk.


NEW YORK – Recent advances in artificial intelligence have been nothing short of dramatic. AI is transforming nearly every sector of society, from transportation to medicine to defense. So it is worth considering what will happen when it becomes even more advanced than it already is.

The apocalyptic view is that AI-driven machines will outsmart humanity, take over the world, and kill us all. This scenario crops up often in science fiction, and it is easy enough to dismiss, given that humans remain firmly in control. But many AI experts take the apocalyptic perspective seriously, and they are right to do so. The rest of society should as well.

To understand what is at stake, consider the distinction between “narrow AI” and “artificial general intelligence” (AGI). Narrow AI can operate only in one or a few domains at a time, so while it may outperform humans in select tasks, it remains under human control.

AGI, by contrast, can reason across a wide range of domains, and thus could replicate many human intellectual skills, while retaining all of the advantages of computers, such as perfect memory recall. Run on sophisticated computer hardware, AGI could outpace human cognition. In fact, it is hard to conceive an upper limit for how advanced AGI could become.

As it stands, most AI is narrow. Indeed, even the most advanced current systems have only limited amounts of generality. For example, while Google DeepMind’s AlphaZero system was able to master Go, chess, and shogi – making it more general than most other AI systems, which can be applied only to a single specific activity – it has still demonstrated capability only within the limited confines of certain highly structured board games.

Many knowledgeable people dismiss the prospect of advanced AGI. Some, such as Selmer Bringsjord of Rensselaer Polytechnic Institute and Drew McDermott of Yale University, argue that it is impossible for AI to outsmart humanity. Others, such as Margaret A. Boden of the University of Sussex and Oren Etzioni of the Allen Institute for Artificial Intelligence, argue that human-level AI may be possible in the distant future, but that it is far too early to start worrying about it now.

These skeptics are not marginal figures, like the cranks who try to cast doubt on climate-change science. They are distinguished scholars in computer science and related fields, and their opinions must be taken seriously.

Yet other distinguished scholars – including David J. Chalmers of New York University, Yale University’s Allan Dafoe and Stuart Russell of the University of California, Berkeley, Nick Bostrom of Oxford University, and Roman Yampolskiy of the University of Louisville – do worry that AGI could pose a serious or even existential threat to humanity. With experts lining up on both sides of the debate, the rest of us should keep an open mind.

Moreover, AGI is the focus of significant research and development. I recently completed a survey of AGI R&D projects, identifying 45 in 30 countries on six continents. Many active initiatives are based in major corporations such as Baidu, Facebook, Google, Microsoft, and Tencent, and in top universities such as Carnegie Mellon, Harvard, and Stanford, as well as the Chinese Academy of Sciences. It would be unwise simply to assume that none of these projects will succeed.

Another way of thinking about the potential threat of AGI is to compare it to other catastrophic risks. In the 1990s, the US Congress saw fit to have NASA track large asteroids that could collide with the earth, even though the odds of that happening are around one in 5,000 per century. With AGI, the odds of a catastrophe over the upcoming century could be as high as one in a hundred, or even one in ten, judging by the pace of R&D and the strength of expert concern.

The question, then, is what to do about it. For starters, we need to ensure that R&D is conducted responsibly, safely, and ethically. This will require a deeper dialogue between those working in the AI field and policymakers, social scientists, and concerned citizens. Those in the field know the technology and will be the ones to design it according to agreed standards; but they must not decide alone what those standards will be. Many of the people developing AI applications are not accustomed to thinking about the social implications of their work. For that to change, they must be exposed to outside perspectives.

Policymakers also will have to grapple with AGI’s international dimensions. Currently, the bulk of AGI R&D is carried out in the United States, Europe, and China, but much of the code is open source, meaning that the work potentially can be done from anywhere. So, establishing standards and ethical ground rules is ultimately a job for the entire international community, though the R&D hubs should take the lead.

Looking ahead, some efforts to address the risks posed by AGI can piggyback on policy initiatives already put in place for narrow AI, such as the new bipartisan AI Caucus launched by John Delaney, a Democratic congressman from Maryland. There are many opportunities for synergy between those working on near-term AI risks and those thinking about the long term.

But regardless of whether narrow AI and AGI are considered together or separately, what matters most is that we take constructive action now to minimize the risk of a catastrophe down the road. This is not a task that we can hope to complete at the last minute.

Seth Baum is the executive director of the Global Catastrophic Risk Institute (GCRI), a think tank focused on extreme global risks. He is also affiliated with the Blue Marble Space Institute of Science and the University of Cambridge Centre for the Study of Existential Risk.

Copyright: Project Syndicate, 2018.


This article is brought to you by Project Syndicate that is a not for profit organization.

Project Syndicate brings original, engaging, and thought-provoking commentaries by esteemed leaders and thinkers from around the world to readers everywhere. By offering incisive perspectives on our changing world from those who are shaping its economics, politics, science, and culture, Project Syndicate has created an unrivalled venue for informed public debate. Please see:

Should you want to support Project Syndicate you can do it by using the PayPal icon below. Your donation is paid to Project Syndicate in full after PayPal has deducted its transaction fee. Facts & Arts neither receives information about your donation nor a commission.



Browse articles by author

More Current Affairs

Nov 12th 2018
It is clearly time for New Deal II. Instead of promising more tax breaks for the richest citizens, a more equitable fiscal policy could pay for necessary bridges and other public goods and services that would improve everyone’s life. Affordable health care for all citizens is a mark of a civilized society. The US is still a long way from that goal. The same is true of high-quality public education. It is grotesque that so many people who stand to benefit from such “socialist” policies are still persuaded to vote against them because they are supposedly “un-American.”
Nov 2nd 2018
The cold-blooded killing of the journalist Khashoggi, however gruesome, pales compared to the brutality and gross human rights violations Saudi Arabia is committing in Yemen. The Saudis are deliberately preventing food and medicine from reaching areas where children are dying from starvation or disease. Their indiscriminate bombings are killing thousands of innocent men, women, and children, leaving whole communities in ruin. The saddest part of this unfolding tragedy is that the US and other Western powers are supplying the Saudis with the weapons they need to massacre the Yemenites, who are trapped in this proxy war between Saudi Arabia and Iran (which neither can win), and the Yemenites will continue to pay with their blood.............Out of a total population of 28 million people, 22 million are in need of humanitarian aid. Nearly 5.2 million children are starving to death, and nearly one million are believed to be infected with cholera. Over 8 million people are facing famine, and 2 million are displaced and deprived of basic needs.
Oct 29th 2018
The nightmare election possibility for the Democrats is continued Republican control of both chambers. In that case, Trump will feel vindicated and more liberated than ever. He might then fire a raft of officials, treat immigrants still more harshly, and try to shut down Mueller’s investigation of his campaign’s possible collusion with the Kremlin and Trump’s probable obstruction of justice. The conventional wisdom may prevail, with the Democrats winning the House but not the Senate. But the polls have been fluctuating. And since Trump’s stunning election victory in 2016, most observers have become more cautious about predicting outcomes.
Oct 23rd 2018
As the Brexit negotiations peter out this week in Brussels, fevered Brexit fanatics – from Boris Johnson, David Davis and Jacob Rees Mogg in the Telegraph, to many others on Twitter – are ranting and raving about the most sensible thing Theresa May has done in two and a half years of Brexit negotiations by suggesting extending the transition period in an attempt at genuine compromise. This would be a good opportunity to remind ourselves of some salient facts. These Conservative MPs are speaking on behalf of the hardest of Brexiteers, a collection of somewhere between 60-80 of the Tory MPs. That’s somewhere between 60 and 80 MPs out of a total of 317 Conservative MPs in the House of Commons. And while having 317 MPs means the Conservatives are the largest party at the last election, they did not win enough of the votes to form a majority. Therefore, for all their bluster and bloviating, let’s just state clearly what the members of this small group are: they are a minority faction, holding a minority view, in a minority government.
Oct 23rd 2018

A billboard at a construction site, with a photo of an Ottoman-style mosque with four minarets and the flag of Turkey, was erected recently in the center of Pristina, the capital of Kosovo.

Oct 17th 2018
Yemen is a country of some 29 million persons, but over a third of them are at risk of starvation if Saudi and UAE bombing campaigns continue.
Oct 14th 2018
Now the Trump administration is eroding the dollar’s global role. Having unilaterally reimposed sanctions on Iran, it is threatening to penalize companies doing business with the Islamic Republic by denying them access to US banks. The threat is serious because US banks are the main source of dollars used in cross-border transactions. According to the Society for Worldwide Interbank Financial Telecommunication (SWIFT), dollars are used in nearly half of all cross-border payments, a share far greater than the weight of the US in the world economy. In response to the Trump administration’s stance, Germany, France, and Britain, together with Russia and China, have announced plans to circumvent the dollar, US banks, and US government scrutiny. “Plans” may be a bit strong, given that few details have been provided. But the three countries have described in general terms the creation of a stand-alone financial entity, owned and organized by the governments in question, to facilitate transactions between Iran and foreign companies.
Oct 5th 2018
There are a lot of oddballs in US President Donald Trump’s entourage, but few are as odd – or as sinister – as 33-year-old Stephen Miller, Trump’s senior policy adviser. Miller resembles a type on the far right that is more common in Europe than the US: young, slick, sharp-suited, even a trifle dandyish. He is a skilled rabble-rouser, whose inflammatory rhetoric against immigrants and refugees – “We’re going to build that wall high and we’re going to build it tall !”– drives the crowds at Trump rallies into a frenzy. One of his crowd-pleasing notions is that migrants will infect Americans with terrible diseases.
Oct 3rd 2018 we are in 2018, 40 years after Camp David. The Palestinian dream of an independent state is not only unrealized but is most likely unrealizable. With many Palestinians now favoring a one state solution......the once "Arab minority"  is now a majority.....
Sep 25th 2018
The US stock market, as measured by the monthly real (inflation-adjusted) S&P Composite Index, or S&P 500, has increased 3.3-fold since its bottom in March 2009. This makes the US stock market the most expensive in the world, according to the cyclically adjusted price-to-earnings (CAPE) ratio that I have long advocated. Is the price increase justified, or are we witnessing a bubble?
Sep 23rd 2018
Global debt recently hit a new record high of 225% of world GDP, amounting to US$164 trillion. The world is now 12 points deeper in debt than the previous peak in 2009, with advanced economies’ ratios at levels not seen since World War II.
Sep 18th 2018
To understand them, it is worth looking at three reputable leaders who died this summer: former United Nations Secretary-General Kofi Annan, former British Foreign Secretary and NATO Secretary-General Peter Carrington, and US Senator John McCain. Having worked with Annan and for Carrington, I can vouch for their grace, honor, and commitment to truth. McCain plainly had the same qualities, not to mention a level of personal bravery far beyond what is expected of most of us (though it should be noted that Carrington was also a war hero). These leaders’ combination of honor and commitment to truth – two attributes that are intrinsically connected – is nowhere to be seen in Trump or Johnson.
Sep 18th 2018
From controlling the media to stoking nationalism, Russian President Vladimir Putin has always known how to keep his approval ratings high. But Russians’ lives are not getting any better, especially after the latest round of Western economic sanctions – and Putin’s declining approval rating shows it.
Sep 15th 2018
As we mark the decennial of the collapse of Lehman Brothers, there are still ongoing debates about the causes and consequences of the financial crisis, and whether the lessons needed to prepare for the next one have been absorbed. But looking ahead, the more relevant question is what actually will trigger the next global recession and crisis, and when. The current global expansion will likely continue into next year, given that the US is running large fiscal deficits, China is pursuing loose fiscal and credit policies, and Europe remains on a recovery path. But by 2020, the conditions will be ripe for a financial crisis, followed by a global recession. There are 10 reasons for this. First, the fiscal-stimulus policies that are currently pushing the annual US growth rate above its 2% potential are unsustainable. By 2020, the stimulus will run out, and a modest fiscal drag will pull growth from 3% to slightly below 2%.
Sep 12th 2018
Next month, a judge in Oregon will begin hearing a case brought against the United States government on behalf of 21 young people, supported by the non-profit organization Our Children’s Trust, who allege that the authorities’ active contributions to the climate crisis violate their constitutional rights. The government defendants have repeatedly tried – so far without success – to have the case thrown out or delayed, and the trial is currently scheduled to start on October 29.
Sep 5th 2018
Wars are expensive, as the Russian people are now learning. The Kremlin is pursuing military adventures in Eastern Ukraine and Syria, and though these conflicts are limited in scope, one wonders if the country can really afford them.
Sep 1st 2018
This week, the California state legislature voted to mandate that all the state’s electricity come from non-carbon sources (chiefly wind, solar and hydro) by 2045. Since California if it were a country would have the world’s fifth largest economy, and since so many other states are economically integrated with it, this plan, if signed by governor Jerry Brown, could help transform the entire country. The goal is less difficult than it seems on the surface. California had already committed to getting one third of its electricity from renewables by 2020, and reached that goal in 2017. It committed to getting 50% of its electricity from renewables by 2030, and in fact will likely reach that goal 10 years early, in 2020.
Aug 29th 2018
Quote: "This may ultimately result in creation of a new accounting standard - the Enterprise Value of Data – which could become an integral part of financial statements, capturing the value of the largest and most ignored corporate asset: data."
Aug 29th 2018
What comes through clearly in polling on US public opinion is that there is a deep partisan divide on the Israeli/Palestinian issue, with key demographic groups increasingly more supportive of Palestinian rights and antagonistic to hardline Israeli policies. In some ways, the Netanyahu/Trump "marriage" has also helped to fuel the partisan divide. A Pew poll from earlier this year found that support for Palestinians far surpasses support for Israel among self-described "progressive" and "liberal" voters. And a recent Gallup poll shows that only 17% of Democrats now have a favorable view of the Israeli leader. 
Aug 27th 2018
History suggests that current-account imbalances ultimately matter a great deal. A still-unbalanced global economy may be forced to relearn that painful lesson in the coming years.