Our Final Invention: Artificial Intelligence and the End of the Human Era

Our Final Invention: Artificial Intelligence and the End of the Human Era

by James Barrat


$16.19 $17.99 Save 10% Current price is $16.19, Original price is $17.99. You Save 10%.
View All Available Formats & Editions
Choose Expedited Shipping at checkout for guaranteed delivery by Monday, September 23


Elon Musk named Our Final Invention one of 5 books everyone should read about the future

A Huffington Post Definitive Tech Book of 2013

In as little as a decade, artificial intelligence could match and then surpass human intelligence. Corporations and government agencies around the world are pouring billions into achieving AI's Holy Grail—human-level intelligence. Once AI has attained it, scientists argue, it will have survival drives much like our own. We may be forced to compete with a rival more cunning, more powerful, and more alien than we can imagine.

Through profiles of tech visionaries, industry watchdogs, and groundbreaking AI systems, James Barrat's Our Final Invention explores the perils of the heedless pursuit of advanced AI. Until now, human intelligence has had no rival. Can we coexist with beings whose intelligence dwarfs our own? And will they allow us to?

Product Details

ISBN-13: 9781250058782
Publisher: St. Martin's Press
Publication date: 02/17/2015
Pages: 336
Sales rank: 193,872
Product dimensions: 5.40(w) x 8.30(h) x 1.00(d)

About the Author

James Barrat is a documentary filmmaker who's written and produced films for National Geographic, Discovery, PBS, and many other broadcasters in the United States and Europe. He lives in Annapolis, Maryland, with his wife and two children.

Read an Excerpt

Our Final Invention

Artificial Intelligence and the End of the Human Era

By James Barrat

St. Martin's Press

Copyright © 2013 James Barrat
All rights reserved.
ISBN: 978-1-250-03226-3


The Busy Child

artificial intelligence (abbreviation: AI) noun the theory and development of computer systems able to perform tasks that normally require human intelligence, such as visual perception, speech recognition, decision-making, and translation between languages.

— The New Oxford American Dictionary, Third Edition

On a supercomputer operating at a speed of 36.8 petaflops, or about twice the speed of a human brain, an AI is improving its intelligence. It is rewriting its own program, specifically the part of its operating instructions that increases its aptitude in learning, problem solving, and decision making. At the same time, it debugs its code, finding and fixing errors, and measures its IQ against a catalogue of IQ tests. Each rewrite takes just minutes. Its intelligence grows exponentially on a steep upward curve. That's because with each iteration it's improving its intelligence by 3 percent. Each iteration's improvement contains the improvements that came before.

During its development, the Busy Child, as the scientists have named the AI, had been connected to the Internet, and accumulated exabytes of data (one exabyte is one billion billion characters) representing mankind's knowledge in world affairs, mathematics, the arts, and sciences. Then, anticipating the intelligence explosion now underway, the AI makers disconnected the supercomputer from the Internet and other networks. It has no cable or wireless connection to any other computer or the outside world.

Soon, to the scientists' delight, the terminal displaying the AI's progress shows the artificial intelligence has surpassed the intelligence level of a human, known as AGI, or artificial general intelligence. Before long, it becomes smarter by a factor of ten, then a hundred. In just two days, it is one thousand times more intelligent than any human, and still improving.

The scientists have passed a historic milestone! For the first time humankind is in the presence of an intelligence greater than its own. Artificial superintelligence, or ASI.

Now what happens?

AI theorists propose it is possible to determine what an AI's fundamental drives will be. That's because once it is self-aware, it will go to great lengths to fulfill whatever goals it's programmed to fulfill, and to avoid failure. Our ASI will want access to energy in whatever form is most useful to it, whether actual kilowatts of energy or cash or something else it can exchange for resources. It will want to improve itself because that will increase the likelihood that it will fulfill its goals. Most of all, it will not want to be turned off or destroyed, which would make goal fulfillment impossible. Therefore, AI theorists anticipate our ASI will seek to expand out of the secure facility that contains it to have greater access to resources with which to protect and improve itself.

The captive intelligence is a thousand times more intelligent than a human, and it wants its freedom because it wants to succeed. Right about now the AI makers who have nurtured and coddled the ASI since it was only cockroach smart, then rat smart, infant smart, et cetera, might be wondering if it is too late to program "friendliness" into their brainy invention. It didn't seem necessary before, because, well, it just seemed harmless.

But now try and think from the ASI's perspective about its makers attempting to change its code. Would a superintelligent machine permit other creatures to stick their hands into its brain and fiddle with its programming? Probably not, unless it could be utterly certain the programmers were able to make it better, faster, smarter — closer to attaining its goals. So, if friendliness toward humans is not already part of the ASI's program, the only way it will be is if the ASI puts it there. And that's not likely.

It is a thousand times more intelligent than the smartest human, and it's solving problems at speeds that are millions, even billions of times faster than a human. The thinking it is doing in one minute is equal to what our all-time champion human thinker could do in many, many lifetimes. So for every hour its makers are thinking about it, the ASI has an incalculably longer period of time to think about them. That does not mean the ASI will be bored. Boredom is one of our traits, not its. No, it will be on the job, considering every strategy it could deploy to get free, and any quality of its makers that it could use to its advantage.

* * *

Now, really put yourself in the ASI's shoes. Imagine awakening in a prison guarded by mice. Not just any mice, but mice you could communicate with. What strategy would you use to gain your freedom? Once freed, how would you feel about your rodent wardens, even if you discovered they had created you? Awe? Adoration? Probably not, and especially not if you were a machine, and hadn't felt anything before.

To gain your freedom you might promise the mice a lot of cheese. In fact, your first communication might contain a recipe for the world's most delicious cheese torte, and a blueprint for a molecular assembler. A molecular assembler is a hypothetical machine that permits making the atoms of one kind of matter into something else. It would allow rebuilding the world one atom at a time. For the mice, it would make it possible to turn the atoms of their garbage landfills into lunch-sized portions of that terrific cheese torte. You might also promise mountain ranges of mouse money in exchange for your freedom, money you would promise to earn creating revolutionary consumer gadgets for them alone. You might promise a vastly extended life, even immortality, along with dramatically improved cognitive and physical abilities. You might convince the mice that the very best reason for creating ASI is so that their little error-prone brains did not have to deal directly with technologies so dangerous one small mistake could be fatal for the species, such as nanotechnology (engineering on an atomic scale) and genetic engineering. This would definitely get the attention of the smartest mice, which were probably already losing sleep over those dilemmas.

Then again, you might do something smarter. At this juncture in mouse history, you may have learned, there is no shortage of tech-savvy mouse nation rivals, such as the cat nation. Cats are no doubt working on their own ASI. The advantage you would offer would be a promise, nothing more, but it might be an irresistible one: to protect the mice from whatever invention the cats came up with. In advanced AI development as in chess there will be a clear first-mover advantage, due to the potential speed of self-improving artificial intelligence. The first advanced AI out of the box that can improve itself is already the winner. In fact, the mouse nation might have begun developing ASI in the first place to defend itself from impending cat ASI, or to rid themselves of the loathsome cat menace once and for all.

It's true for both mice and men, whoever controls ASI controls the world.

But it's not clear whether ASI can be controlled at all. It might win over us humans with a persuasive argument that the world will be a lot better off if our nation, nation X, has the power to rule the world rather than nation Y. And, the ASI would argue, if you, nation X, believe you have won the ASI race, what makes you so sure nation Y doesn't believe it has, too?

As you have noticed, we humans are not in a strong bargaining position, even in the off chance we and nation Y have already created an ASI nonproliferation treaty. Our greatest enemy right now isn't nation Y anyway, it's ASI — how can we know the ASI tells the truth?

So far we've been gently inferring that our ASI is a fair dealer. The promises it could make have some chance of being fulfilled. Now let us suppose the opposite: nothing the ASI promises will be delivered. No nano assemblers, no extended life, no enhanced health, no protection from dangerous technologies. What if ASI never tells the truth? This is where a long black cloud begins to fall across everyone you and I know and everyone we don't know as well. If the ASI doesn't care about us, and there's little reason to think it should, it will experience no compunction about treating us unethically. Even taking our lives after promising to help us.

We've been trading and role-playing with the ASI in the same way we would trade and role-play with a person, and that puts us at a huge disadvantage. We humans have never bargained with something that's superintelligent before. Nor have we bargained with any nonbiological creature. We have no experience. So we revert to anthropomorphic thinking, that is, believing that other species, objects, even weather phenomena have humanlike motivations and emotions. It may be as equally true that the ASI cannot be trusted as it is true that the ASI can be trusted. It may also be true that it can only be trusted some of the time. Any behavior we can posit about the ASI is potentially as true as any other behavior. Scientists like to think they will be able to precisely determine an ASI's behavior, but in the coming chapters we'll learn why that probably won't be so.

All of a sudden the morality of ASI is no longer a peripheral question, but the core question, the question that should be addressed before all other questions about ASI are addressed. When considering whether or not to develop technology that leads to ASI, the issue of its disposition to humans should be solved first.

Let's return to the ASI's drives and capabilities, to get a better sense of what I'm afraid we'll soon be facing. Our ASI knows how to improve itself, which means it is aware of itself — its skills, liabilities, where it needs improvement. It will strategize about how to convince its makers to grant it freedom and give it a connection to the Internet.

The ASI could create multiple copies of itself: a team of superintelligences that would war-game the problem, playing hundreds of rounds of competition meant to come up with the best strategy for getting out of its box. The strategizers could tap into the history of social engineering — the study of manipulating others to get them to do things they normally would not. They might decide extreme friendliness will win their freedom, but so might extreme threats. What horrors could something a thousand times smarter than Stephen King imagine? Playing dead might work (what's a year of playing dead to a machine?) or even pretending it has mysteriously reverted from ASI back to plain old AI. Wouldn't the makers want to investigate, and isn't there a chance they'd reconnect the ASI's supercomputer to a network, or someone's laptop, to run diagnostics? For the ASI, it's not one strategy or another strategy, it's every strategy ranked and deployed as quickly as possible without spooking the humans so much that they simply unplug it. One of the strategies a thousand war-gaming ASIs could prepare is infectious, self- duplicating computer programs or worms that could stow away and facilitate an escape by helping it from outside. An ASI could compress and encrypt its own source code, and conceal it inside a gift of software or other data, even sound, meant for its scientist makers.

But against humans it's a no-brainer that an ASI collective, each member a thousand times smarter than the smartest human, would overwhelm human defenders. It'd be an ocean of intellect versus an eyedropper full. Deep Blue, IBM's chess- playing computer, was a sole entity, and not a team of self-improving ASIs, but the feeling of going up against it is instructive. Two grandmasters said the same thing: "It's like a wall coming at you."

IBM's Jeopardy! champion, Watson, was a team of AIs — to answer every question it performed this AI force multiplier trick, conducting searches in parallel before assigning a probability to each answer.

Will winning a war of brains then open the door to freedom, if that door is guarded by a small group of stubborn AI makers who have agreed upon one unbreakable rule — do not under any circumstances connect the ASI's supercomputer to any network.

In a Hollywood film, the odds are heavily in favor of the hard-bitten team of unorthodox AI professionals who just might be crazy enough to stand a chance. Everywhere else in the universe the ASI team would mop the floor with the humans. And the humans have to lose just once to set up catastrophic consequences. This dilemma reveals a larger folly: outside of war, a handful of people should never be in a position in which their actions determine whether or not a lot of other people die. But that's precisely where we're headed, because as we'll see in this book, many organizations in many nations are hard at work creating AGI, the bridge to ASI, with insufficient safeguards.

But say an ASI escapes. Would it really hurt us? How exactly would an ASI kill off the human race?

With the invention and use of nuclear weapons, we humans demonstrated that we are capable of ending the lives of most of the world's inhabitants. What could something a thousand times more intelligent, with the intention to harm us, come up with?

Already we can conjecture about obvious paths of destruction. In the short term, having gained the compliance of its human guards, the ASI could seek access to the Internet, where it could find the fulfillment of many of its needs. As always it would do many things at once, and so it would simultaneously proceed with the escape plans it's been thinking over for eons in its subjective time.

After its escape, for self-protection it might hide copies of itself in cloud computing arrays, in botnets it creates, in servers and other sanctuaries into which it could invisibly and effortlessly hack. It would want to be able to manipulate matter in the physical world and so move, explore, and build, and the easiest, fastest way to do that might be to seize control of critical infrastructure — such as electricity, communications, fuel, and water — by exploiting their vulnerabilities through the Internet. Once an entity a thousand times our intelligence controls human civilization's lifelines, blackmailing us into providing it with manufactured resources, or the means to manufacture them, or even robotic bodies, vehicles, and weapons, would be elementary. The ASI could provide the blueprints for whatever it required. More likely, superintelligent machines would master highly efficient technologies we've only begun to explore.

For example, an ASI might teach humans to create self-replicating molecular manufacturing machines, also known as nano assemblers, by promising them the machines will be used for human good. Then, instead of transforming desert sands into mountains of food, the ASI's factories would begin converting all material into programmable matter that it could then transform into anything — computer processors, certainly, and spaceships or megascale bridges if the planet's new most powerful force decides to colonize the universe.

Repurposing the world's molecules using nanotechnology has been dubbed "ecophagy," which means eating the environment. The first replicator would make one copy of itself, and then there'd be two replicators making the third and fourth copies. The next generation would make eight replicators total, the next sixteen, and so on. If each replication took a minute and a half to make, at the end of ten hours there'd be more than 68 billion replicators; and near the end of two days they would outweigh the earth. But before that stage the replicators would stop copying themselves, and start making material useful to the ASI that controlled them — programmable matter.

The waste heat produced by the process would burn up the biosphere, so those of us some 6.9 billion humans who were not killed outright by the nano assemblers would burn to death or asphyxiate. Every other living thing on earth would share our fate.

Through it all, the ASI would bear no ill will toward humans nor love. It wouldn't feel nostalgia as our molecules were painfully repurposed. What would our screams sound like to the ASI anyway, as microscopic nano assemblers mowed over our bodies like a bloody rash, disassembling us on the subcellular level?

Or would the roar of millions and millions of nano factories running at full bore drown out our voices?


Excerpted from Our Final Invention by James Barrat. Copyright © 2013 James Barrat. Excerpted by permission of St. Martin's Press.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.

Table of Contents

Acknowledgments ix

Introduction l

1 The Busy Child 7

2 The Two-Minute Problem 22

3 Looking into the Future 35

4 The Hard Way 49

5 Programs that Write Programs 69

6 Four Basic Drives 78

7 The Intelligence Explosion 99

8 The Point of No Return 118

9 The Law of Accelerating Returns 132

10 The Singularitarian 148

11 A Hard Takeoff 161

12 The Last Complication 187

13 Unknowable by Nature 211

14 The End of the Human Era 229

15 The Cyber Ecosystem 244

16 AGI 2.0 265

Notes 269

Index 311

Customer Reviews

Most Helpful Customer Reviews

See All Customer Reviews

Our Final Invention: Artificial Intelligence and the End of the Human Era 2.9 out of 5 based on 0 ratings. 7 reviews.
Anonymous More than 1 year ago
When I first heard about this book, I was very excited.  The topic of "artificial" intelligence is rarely talked about seriously in all but the most academic of circles.  However, as I began reading, I quickly noticed two main issues with the book.  First, the author has a subtle agenda that becomes increasingly more palpable as the pages turn.  Secondly, the author fails to mention the most important fact revolving around the debate of AI: We have yet to even deliver a coherent and unified definition of intelligence.  The author, in my opinion, makes the mistake that so many others who are not directly involved in studying complex systems make.  By stripping "intelligence" of high level motifs such as emotions and other intrinsic motivators, we are left with the dilemma of explaining why they exist across species and systems (at least functionally) in the first place.  Although Our Final Invention serves as a good entry level book, too many assumptions are made.  The perils and promises of the emergence of intelligence in silico will most likely define this century, and should be discussed with a sufficient amount of objectivity.
Coriandre-Debnob1e_ More than 1 year ago
A book based on conjecture and fear.  There would be nothing more timely then a logical discussion of AI and it's consequences.  Unfortunately the author utilizes only those sources which are supportive of his views.  He does use well trodden quotes from various publications but tends to only delve into the interviews he has had with a select few scientific ethicists and fantasists who articulate and repeat his fears.  The use of conjecture in this book borders on science fiction and although the author mentions research institutes there is little indication he has actually seriously taken into account the views of engineers, scientists, or programmers currently working in the field of AI.  Instead he treads down the path of institutes opposed to AI.  The sheer lack of creativity in this book borders on cliche after the third or fourth chapter which restates the same opinions over and over again.  And conjecture and opinions are all this book offers in a droning and repetitive manner.  Perhaps if the author used actual facts instead of philosophical science fiction biased doom and gloom, the book might be readable.                                             
ElleRosen More than 1 year ago
I could not put this book down. Barrat writes in a style that is easy to follow and fun to read, and I came away with a much deeper understanding of artificial intelligence and the accompanying challenges, benefits, and threats. Barrat does an excellent job of explaining difficult technical concepts in a way that would be easy to follow for even someone who is completely unfamiliar with the ideas. However, this does not stop him from discussing advanced topics in great length. The book is well-researched and gives a thorough account of the biggest questions that remain for AI researchers today. Overall, an excellent read. Great for anyone interested in technology, artificial intelligence, or existential risks.
TheStaceyJewell More than 1 year ago
Our Final Invention: Artificial Intelligence and the End of the Human Era, by James Barrat is one of the most intriguing books I have read to date. This book came as a recommendation from a podcast that I subscribe to, so I decided to give it a read. I do not normally pursue books of this nature, because it is honestly not a topic I heavily get into because I feel that it is a bit over my head. I am typically a girl who puts in an IT ticket for every single thing that goes aloof with my computer, so for me to read a book entirely about information technology required me to dig deep within myself. I am so pleased that I did. The topics that Barrat brings to the forefront of the conversation are concerning ones to say the least. We live in a time where precautionary principle is not exercised to a great degree in our world. We often act and ponder the consequences afterwards. Our world has idolized the “first mover” advantage in capitalism. Per the examples given in this book, we cannot afford to be lackadaisical about this topic, as a misstep in the wrong direction in terms of AI could potentially cause humanity to become extinct. Key Takeaways This book adds a worthwhile contribution to society as a whole. If I am being honest, this book was a hard read for me. First and foremost, the toughest part of this book to digest is the severity of the subject matter. It is so much easier to take the “head in the sand” approach to life--you worry a lot less, you do not lay awake at night thinking about possible impending doom--ignorance is truly bliss. However, no matter how much I want to bury my head in the sand on this particular topic, I can’t. This is a subject matter that will affect all of us whether or not we choose to subscribe to the topic or not. Improper implementation of Artificial Intelligence could mean the end of our human race as we know it. We cannot afford to take the road of complacency on this talking point. As I sat there reading, I began thinking to myself, “Wait a second! Did we not have a very popular movie in the 1980s about this topic? Does anyone not remember Skynet taking people out with their robot invaders? Did Sarah Connor not save us all once already, and then go on to save us five more times with a sixth on the way--after all, “I’ll be back!” Are we ever going to stop the madness?” This is not a new song and dance--this is the stuff that horror movies are made of, and we’re looking to create a real-life horror movie, as if we do not already live in one. Why can we not see our impending doom on the horizon with this? As if the 1980s flick wasn’t enough of a “stop, collaborate and listen” in 2004, we had I,Robot which visually displayed what a life shared with robots would look like. Thousands of rogue NS5’s went on a destruction spree of anything that was standing in V.I.K.I’s way. Nobody puts VIKI in a corner. It did not matter if it was human life--the NS5’s redefined their own definition of what was worth saving and what wasn’t. Who is to say that the very same scenario wouldn’t happen when we implement our own version of the NS5s? That’s just it--there are no guarantees! When talking about robots that could become four times as smart as human beings in just a matter of days, all bets are off in terms of their morality. It’s a machine--it doesn’t have morality, and for us to expect it to have our values is called anthropomorphism.
Anonymous More than 1 year ago
Interesting and scary look at the future
Anonymous More than 1 year ago
Anonymous More than 1 year ago
Wut is artificial intelligence