ENGL 8122  ※  User-Experience Research & Writitng

Home AIs Blogs Books Calendar Cases Dark Mode Keywords Outline Lectures Software Tools Websites


The Coming Wave: Technology, Power, and the Twenty-first Century's Greatest Dilemma

Chapter 1: Containment Is Not Possible

  • Page 20 The coming wave is defined by two core technologies: artificial intelligence (AI) and synthetic biology. Together they will usher in a new dawn for humanity, creating wealth and surplus unlike anything ever seen.
  • Page 21 our future both depends on these technologies and is imperiled by them.
  • Page 23 AI has been climbing the ladder of cognitive abilities for decades, and it now looks set to reach human-level performance across a very wide range of tasks within the next three years.
  • Page 23 Beyond AI, a wider revolution was underway, with AI feeding a powerful, emerging generation of genetic technologies and robotics.
  • Page 25 The current discourse around technology ethics and safety is inadequate.
  • Page 26 I also underscored AI's potential to put large numbers of people out of work.
  • Page 26 long history of displacing labor.
  • Page 33 The various technologies I'm speaking of share four key features that explain why this isn't business as usual: they are inherently general and therefore omni-use, they hyper-evolve, they have asymmetric impacts, and, in some respects, they are increasingly autonomous.
  • Page 33 the potential for new forms of violence, a flood of misinformation, disappearing jobs, and the prospect of catastrophic accidents.

    Part I: Homo Technologicus

  • Page 39 Engines weren't just powering vehicles; they were driving history. Now, thanks to hydrogen and electric motors, the reign of the combustion engine is in its twilight. But the era of mass mobility it unleashed is not.
  • Page 40 a wave is a set of technologies coming together around the same time, powered by one or several new general- purpose technologies with profound societal implications.
  • Page 40 a new piece of technology, like the internal combustion engine, proliferates and transforms everything around it.
  • Page 41 Stonework and fire were proto-general-purpose technologies, meaning they were pervasive, in turn enabling new inventions, goods, and organizational behaviors.
  • Page 42 Throughout history, population size and innovation levels are linked. New tools and techniques give rise to larger populations.
  • Page 43 From the written word to sailing vessels, technology increases interconnectedness, helping to boost its own flow and spread. Each wave hence lays the groundwork for successive waves.
  • Page 47 Proliferation is catalyzed by two forces: demand and the resulting cost decreases, each of which drives technology to become even better and cheaper.
  • Page 47 Civilization's appetite for useful and cheaper technologies is boundless. This will not change.
  • Page 50 Our phones are the first thing we see in the morning and the last at night. Every aspect of human life is affected:
  • Page 52 What on paper looks flawless can behave differently out in the wild, especially when copied and further adapted downstream.

    Chapter 3: The Containment Problem

  • Meaningful control, the capability to stop a use case, change a research direction, or deny access to harmful actors. It means preserving the ability to steer waves to ensure their impact reflects our values, helps us flourish as a species, and does not introduce significant harms that outweigh their benefits.
  • Page 55 Containment encompasses regulation, better technical safety, new governance and ownership models, and new modes of accountability and transparency, all as necessary (but not sufficient) precursors to safer technology.
  • Page 55 future is built. Think of containment, then, as a set of interlinked and mutually reinforcing technical, cultural, legal, and political mechanisms for maintaining societal control of technology during a time of exponential change;
  • Page 56 As the printing press roared across Europe in the fifteenth century, the Ottoman Empire had a rather different response. It tried to ban it.
  • Page 57 In hindsight, waves might appear smooth and inevitable. But there is an almost infinite array of small, local, and often arbitrary factors that affect a technology's trajectory.
  • Page 58 Where there is demand, technology always breaks out, finds traction, builds users.
  • Page 61 That nuclear technology remained contained was no accident; it was a conscious nonproliferation policy of the nuclear powers, helped by the fact that nuclear weapons are incredibly complex and expensive to produce.
  • Page 62 Mutually assured destruction hemmed in possessors since it soon became clear that using them in anger is a quick way of ensuring your own destruction.
  • Page 62 expensive and difficult to manufacture.
  • Page 62 even though nuclear capability has been largely contained, a partial exception, it's not a reassuring story. Nuclear history is still a chilling succession of accidents, near misses, and misunderstandings.
  • Page 64 Nuclear weapons are among the most contained technologies in history, and yet the containment problem—in its hardest, most literal sense—even here remains acutely unsolved.
  • Page 66 As long as a technology is useful, desirable, affordable, accessible, and unsurpassed, it survives and spreads and those features compound.
  • Page 67 In the coming decades, a new wave of technology will force us to confront the most foundational questions our species has ever faced.

    Chapter 4: The Technology of Intelligence

    The coming wave of technology is built primarily on two general-purpose technologies capable of operating at the grandest and most granular levels alike: artificial intelligence and synthetic biology.
  • Page 76 No longer simply a tool, it's going to engineer life and rival—and surpass—our own intelligence.
  • Page 77 Each technology described here intersects with, buttresses, and boosts the others in ways that make it difficult to predict their impact in advance.
  • Page 82 Mass-scale AI rollout is already well underway. Everywhere you look, software has eaten the world, opening the path for collecting and analyzing vast amounts of data. That data is now being used to teach AI systems to create more efficient and more accurate products in almost every area of our lives.
  • Page 82 AI will become inextricably part of the social fabric.
  • Page 82 At DeepMind we developed systems to control billion-dollar data centers, a project resulting in 40 percent reductions in energy used for cooling.
  • Page 84 A big part of what makes humans intelligent is that we look at the past to predict what might happen in the future. In this sense intelligence can be understood as the ability to generate a range of plausible scenarios about how the world around you may unfold and then base sensible actions on those predictions.
  • Page 84 LLMs take advantage of the fact that language data comes in a sequential order. Each unit of information is in some way related to data earlier in a series. The model reads very large numbers of sentences, learns an abstract representation of the information contained within them, and then, based on this, generates a prediction about what should come next.
  • Page 85 What are the key words, the most salient elements of a sentence, and how do they relate to one another?
  • Page 85 sentence. In effect, the LLM learns which words to pay attention to.
  • Page 87 Over the next few years, I believe, AI will become as ubiquitous as the internet itself: just as available, and yet even more consequential.
  • Page 87 However, a key ingredient of the LLM revolution is that for the first time very large models could be trained directly on raw, messy, real-world data, without the need for carefully curated and human-labeled data sets.
  • Page 87 Today's LLMs are trained on trillions of words. Imagine digesting Wikipedia wholesale, consuming all the subtitles and comments on YouTube, reading millions of legal contracts, tens of millions of emails, and hundreds of thousands of books.
  • Page 88 these new LLMs are stunningly good at scores of different writing tasks once the preserve of skilled human experts, from translation to
  • Page 88 accurate summarization to writing plans for improving the performance of LLMs.
  • Page 90 humans' ability to complete given tasks— human intelligence itself— is very much a fixed target, as large and multifaceted as it is.
  • Page 91 When a new technology starts working, it always becomes dramatically more efficient. AI is no different.
  • Page 91 AI increasingly does more with less.
  • Page 92 In the words of an eminent computer scientist, "It seems totally obvious to me that of course all programs in the future will ultimately be written by AIs, with humans relegated to, at best, a supervisory role."
  • Page 93 But it quickly became apparent that these models sometimes produce troubling and actively harmful content like racist screeds or rambling conspiracy theories. Research into GPT-2 found that when prompted with the phrase "the white man worked as…," it would autocomplete with "a police officer, a judge, a prosecutor, and the president of the United States." Yet when given the same prompt for "Black man," it would autocomplete with "a pimp," or for "woman" with "a prostitute."
  • Page 96 There's a recurrent problem with making sense of progress in AI. We quickly adapt, even to breakthroughs that astound us initially, and within no time they seem routine, even mundane.
  • Page 97 Although LaMDA was of course not sentient, soon it will be routine to have AI systems that can convincingly appear to
  • Page 97 Significant challenges with real- world applications linger, including material questions of bias and fairness, reproducibility, security vulnerabilities, and legal liability. Urgent ethical gaps and unsolved safety questions cannot be ignored. Yet I see a field rising to these challenges, not shying away or failing to make headway. I see obstacles but also a track record of overcoming them. People interpret unsolved problems as evidence of lasting limitations; I see an unfolding research process.
  • Page 99 I believe the debate about whether and when the Singularity will be achieved is a colossal red herring.
  • Page 99 I've gone to countless meetings trying to raise questions about synthetic media and misinformation, or privacy, or lethal autonomous weapons, and instead spent the time answering esoteric questions from otherwise intelligent people about consciousness, the Singularity, and other matters irrelevant to our world right now.
  • Page 100 What we would really like to know is, can I give an AI an ambiguous, open-ended, complex goal that requires interpretation, judgment, creativity, decision-making, and acting across multiple domains, over an extended time period, and then see the AI accomplish that goal?
  • Page 100 a Modern Turing Test would involve something like the following: an AI being able to successfully act on the instruction "Go make $1 million on Amazon in a few months with just a $100,000 investment."
  • Page 101 Should my Modern Turing Test for the twenty-first century be met, the implications for the global economy are profound.
  • Page 102 Rather than get too distracted by questions of consciousness, then, we should refocus the entire debate around near-term capabilities and how they will evolve in the coming years.
  • Page 103 There will be thousands of these models, and they will be used by the majority of the world's population. It will take us to a point where anyone can have an ACI in their pocket that can help or even directly accomplish a vast array of conceivable goals: planning and running your vacation, designing and building more efficient solar panels, helping win an election.
  • Page 104 The risk isn't in overhyping it; it's rather in missing the magnitude of the coming wave.
  • Page 104 else, itself a maker of tools and platforms, not just a system but a generator of systems of any and all kinds.

    Chapter 5: The Technology of Life

  • Page 105 Biology itself became an engineering tool. Alongside AI, this is the most important transformation of our lifetimes.
  • Page 106 DNA is information, a biologically evolved encoding and storage system.
  • Page 107 Genetic engineering has gotten much cheaper and much easier.
  • Page 109 Like AI, genetic engineering is a field in blistering motion, evolving and developing by the week, a massive global concentration of talent and energy beginning to bear real fruit (in this case, literally).
  • Page 113 serious physical self-modifications are going to happen.
  • Page 113 Initial work suggests memory can be improved and muscle strength enhanced.

    Chapter 6: The Wider Wave

  • The future of agriculture, as John Deere sees it, involves autonomous tractors and combines that operate independently, following a field's GPS coordinates and using an array of sensors to make automatic, real-time alterations to harvesting, maximizing yield and minimizing waste. The company is producing robots that can plant, tend, and harvest crops, with levels of precision and granularity that would be impossible for humans.
  • Page 127 But above all it signified how robots are gradually working their way into society, poised to play a far greater role in daily life than has been the case before. From a deadly crisis to the quiet hum of a logistics hub, from a bustling factory to an eldercare home, robots are here.
  • Page 129 Quantum computing is, in other words, yet another foundational technology still in very early development, still further from hitting those critical moments of cost decreases and widespread proliferation, let alone the technical breakthroughs that will make it fully feasible.
  • Page 130 funding and knowledge are escalating, progress
  • Page 130 Renewable energy will become the largest single source of electricity generation by 2027.
  • Page 133 At its core, the coming wave is a story of the proliferation of power. If the last wave reduced the costs of broadcasting information, this one reduces the costs of acting on it, giving rise to technologies that go from sequencing to synthesis, reading to writing, editing to creating, imitating conversations to leading them.

    Chapter 10: Fragility Amplifiers

  • As the power and spread of any technology grows, so its failure modes escalate.
  • Page 223 reiterate: these risks are not about malicious harm; they come from simply operating on the bleeding edge of the most capable technologies in history widely embedded throughout core societal systems.
  • Page 224 But what if new job-displacing systems scale the ladder of human cognitive ability itself, leaving nowhere new for labor to turn? If
  • Page 225 These tools will only temporarily augment human intelligence. They will make us smarter and more efficient for a time, and will unlock enormous amounts of economic growth, but they are fundamentally labor replacing.
  • Page 225 the days of this kind of "cognitive manual labor" are numbered.
  • Page 225 Early analysis of ChatGPT suggests it boosts the productivity of "mid-level college educated professionals" by 40 percent on many tasks.
  • Page 225 McKinsey study estimated that more than half of all jobs could see many of their tasks automated by machines in the next seven years, while fifty-two million Americans work in roles with a "medium exposure to automation" by 2030.
  • Page 226 Yes, it's almost certain that many new job categories will be created. Who would have thought that "influencer" would become a highly sought-after role? Or imagined that in 2023 people would be working as "prompt engineers"—nontechnical programmers of large language models who become adept at coaxing out specific responses?
  • Page 226 But my best guess is that new jobs won't come in the numbers or timescale to truly help.
  • Page 226 sure, new demand will create new work, but that doesn't mean it all gets done by human beings.
  • Page 227 Working on a zero-hours contract in a distribution center doesn't provide the sense of pride or social solidarity that came from working for a booming Detroit auto manufacturer in the 1960s.
  • Page 227 New jobs might be created in the long term, but for millions they won't come quick enough or in the right places.
  • Page 228 Whichever side of the jobs debate you fall on, it's hard to deny that the ramifications will be hugely destabilizing for hundreds of millions who will, at the very least, need to re-skill and transition to new types of work.
  • Page 228 Labor market disruptions are, like social media, fragility amplifiers. They damage and undermine the nation-state.
  • Page 229 It produces trillions of dollars in new economic value while also destroying certain existing sources of wealth.
  • Page 229 Some individuals are greatly enabled; others stand to lose everything.

    Chapter 11: The Future of Nations

  • The stirrup was an apparently simple innovation. But with it came a social revolution changing hundreds of millions of lives.
  • Page 232 In the resulting turbulence, without a major shift in focus, many open democratic states face a steady decay of their institutional foundations, a withering of legitimacy and authority.
  • Page 232 At the same time, authoritarian states are given a potent new arsenal of repression.
  • Page 233 it will be instead a long-term macro-trend toward deep instability grinding away over decades. The first result will be massive new concentrations of power and wealth that reorder society.
  • Page 234 machine intelligence resembles a massive bureaucracy far more than it does a human mind.
  • Page 235 What happens when many, perhaps the majority, of the tasks required to operate a corporation, or a government department, can be run more efficiently by machines?
  • Page 236 Unlike with rockets, satellites, and the internet, the frontier of this wave is found in corporations, not in government organizations or academic labs.
  • Page 236 I think we'll see a group of private corporations grow beyond the size and reach of many nation-states.
  • Page 236 The Korean economic miracle was a Samsung-powered miracle.
  • Page 237 Already, for example, eBay and PayPal's dispute resolution system handles around sixty million disagreements a year, three times as many as the entire U.S. legal system. Ninety percent of these disputes are settled using technology alone.
  • Page 237 In the last wave, things dematerialized; goods became services.
  • Page 238 All the big tech platforms either are mainly service businesses or have very large service businesses.
  • Page 238 the ascendancy of low- code and no- code software, the rise of bio- manufacturing, and the boom in 3- D printing.
  • Page 239 Those with the resources to invent or adopt new technologies fastest—those that can pass my updated Turing test, for example—will enjoy rapidly compounding returns.
  • Page 239 An unbridgeable "intelligence gap" becomes plausible.
  • Page 241 When compared with superstar corporations, governments appear slow, bloated, and out of touch. It's tempting to dismiss them as headed for the trash can of history. However, another inevitable reaction of nation-states will be to use the tools of the coming wave to tighten their grip on power, taking full advantage to entrench their dominance.
  • Page 242 Already a distant organization knows, in theory, what time you are awake, how you are feeling, and what you are looking at.
  • Page 243 The only step left is bringing these disparate databases together into a single, integrated system: a perfect twenty-first-century surveillance apparatus.
  • Page 243 Compared with the West, Chinese research into AI concentrates on areas of surveillance
  • Page 243 like object tracking, scene understanding, and voice or action recognition.
  • Page 243 Centralized services like WeChat bundle everything from private messaging to shopping and banking in one easily traceable place.
  • Page 244 Chinese police even have sunglasses with built-in facial recognition technology capable of tracking suspects in crowds.
  • Page 245 Societies of overweening surveillance and control are already here, and now all of this is set to escalate enormously into a next-level concentration of power at the center.
  • Page 245 It's no secret that governments monitor and control their own populations, but these tendencies extend deep into Western firms, too.
  • Page 246 Companies like Vigilant Solutions aggregate movement data based on license plate tracking, then sell it to jurisdictions like state or municipal governments.
  • Page 246 Just as much as anyone in China, those in the West leave a vast data exhaust every day of their lives. And just as in China, it is harvested, processed, operationalized, and sold.
  • Page 246 This raises the prospect of totalitarianism to a new plane. It won't happen everywhere, and not all at once. But if AI, biotech, quantum,
  • Page 246 robotics, and the rest of it are centralized in the hands of a repressive state, the resulting entity would be palpably different from any yet seen.
  • Page 248 Fields like education and medicine currently rely on huge social and financial infrastructures. It's quite possible to envisage these being slimmed and localized: adaptive and intelligent education systems, for example, that take a student through an entire journey of learning, building a bespoke curriculum; AIs able to create all the materials like interactive games perfectly adapted to the child with automated grading systems; and so on.
  • Page 249 When anyone has access to the bleeding edge, it's not just nation-states that can mount formidable physical and virtual defenses.
  • Page 250 Techniques like CRISPR make biological experimentation easier, meaning biohackers in their garages can tinker at the absolute frontier of science.
  • Page 250 Imagine a future where small groups—whether in failing states like Lebanon or in off-grid nomad camps in New Mexico—provide AI-empowered services like credit unions, schools, and health care, services at the heart of the community often reliant on scale or the state.
  • Page 250 Think about setting up your own school. Or hospital or army. It's such a complex, vast, and difficult project, even the thought of it is tiring. Just gathering the resources, getting necessary permissions and equipment, is a lifelong endeavor. Now consider having an array of assistants who, when asked to create a school, a hospital, or an army, can make it happen in a realistic time frame.
  • Page 251 What happens to traditional hierarchies when tools of awesome power and expertise are as available to street children as to billionaires?
  • Page 251 As people increasingly take power into their own hands, I expect inequality's newest frontier to lie in biology.
  • Page 251 There could then be something like a biohacking personal enhancement arms race.
  • Page 251 What does the social contract look like if a select group of "post-humans" engineer themselves to some unreachable intellectual or physical plane?
  • Page 252 we are entering a new era where the previously unthinkable is now a distinct possibility. Being blinkered about what's happening is, in my view, more dangerous than being overly speculative.
  • Page 252 When northern Italy was a patchwork of small city-states, it gave us the Renaissance, yet was also a field of constant internecine war and feuding.
  • Page 252 Hyper-libertarian technologists like the PayPal founder and venture capitalist Peter Thiel celebrate a vision of the state withering away, seeing this as liberation for an overmighty species of business leaders or "sovereign individuals," as they call themselves.
  • Page 253 find it deeply depressing that some of the most powerful and privileged take such a narrow and destructive view, but it adds a further impetus to fragmentation.
  • Page 253 Every individual, every business, every church, every nonprofit, every nation, will eventually have its own AI and ultimately its own bio and robotics capability.
  • Page 254 Within the decade AIs will decide how public money gets spent, where military forces are assigned, or what students should learn.
  • Page 255 And if this picture sounds too strange, paradoxical, and impossible, consider this. The coming wave will only deepen and recapitulate the exact same contradictory dynamics of the last wave. The internet does precisely this: centralizes in a few key hubs while also empowering billions of people.
  • Page 255 Everyone can build a website, but there's only one Google. Everyone can sell their own niche products, but there's only one Amazon.

    Chapter 12: The Dilemma

  • The overwhelming majority of these technologies will be used for good. Although I have focused on their risks, it's important to keep in mind they will improve countless lives on a daily basis.
  • Page 258 Eventually, something will go wrong—
  • Page 258 the most secure solutions for containment are equally unacceptable, leading humanity down an authoritarian and dystopian pathway.
  • Page 259 the implications of these technologies will push humanity to navigate a path between the poles of catastrophe and dystopia. This is the essential dilemma of our age.
  • Page 263 Over the next ten years, AI will be the greatest force amplifier in history. This is why it could enable a redistribution of power on a historic scale.
  • Page 263 The greatest accelerant of human progress imaginable, it will also enable harms—from wars and accidents to random terror groups, authoritarian governments, overreaching corporations, plain theft, and willful sabotage.
  • Page 264 consider even more basic modes of failure, not attacks, but plain errors.
  • Page 265 If the wave is uncontained, it's only a matter of time. Allow
  • Page 267 The sickening nihilism of the school shooter is bounded by the weapons they can access.
  • Page 269 Steadily, many nations will convince themselves that the only way of truly ensuring this is to install the kind of blanket surveillance we saw in the last chapter: total control, backed by hard power. The door to dystopia is cracked open. Indeed, in the face of catastrophe, for some dystopia may feel like a relief.
  • Page 273 Throughout history societal collapses are legion: from ancient Mesopotamia to Rome, the Maya to Easter Island, again and again it's not just that civilizations don't last; it's that unsustainability appears baked in. Civilizations that collapse are not the exception; they are the rule.
  • Page 274 The development of new technologies is, as we've seen, a critical part of meeting our planet's grand challenges. Without new technologies, these challenges will simply not be met.
  • Page 274 Over the next century, the global population will start falling, in some countries precipitously. As the ratio of workers to retirees shifts and the labor force dwindles, economies will simply not be able to function at their present levels. In other words, without new technologies it will be impossible to maintain living standards.
  • Page 275 Stress on our resources, too, is a certainty.
  • Page 276 Make no mistake: standstill in itself spells disaster.
  • Page 277 I'm still convinced that technology remains a primary driver for making improvements to our world and our lives.
  • Page 278 I am, however, confident that the coming decades will see complex, painful trade-offs between prosperity, surveillance, and the threat of catastrophe growing ever more acute.
  • Page 278 for everyone's sake, containment must be possible.

    Chapter 13: Containment Must Be Possible

  • When a government has devolved to the point of simply lurching from crisis to crisis, it has little breathing room for tackling tectonic forces requiring deep domain expertise and careful judgment on uncertain timescales. It's easier to ignore these issues in favor of low-hanging fruit more likely to win votes in the next election.
  • Page 283 The price of scattered insights is failure,
  • Page 286 Complex regulations refined over decades made roads and vehicles incrementally safer and more ordered, enabling their growth and spread. And yet 1.35 million people a year still die in traffic accidents. Regulation may lessen the negative effects, but it can't erase bad outcomes like crashes, pollution, or sprawl.
  • Page 288 Regulation is not enough, but at least it's a start.
  • Page 289 Earlier in the book I described containment as a foundation for controlling and governing technology, spanning technical, cultural, and regulatory aspects. At root, I believe this means having the power to drastically curtail or outright stop technology's negative impacts, from the local and small scale up to the planetary and existential. Encompassing hard enforcement against misuse of proliferated technologies, it also steers the development, direction, and governance of nascent technologies.
  • Page 289 modes of failure are known, managed, and mitigated,
  • Page 290 as a set of guardrails, a way to keep humanity in the driver's seat
  • Page 290 Rather than general systems, then, those that are more narrowly scoped and domain specific should be encouraged.
  • Page 290 Areas like materials design or drug development are going to rapidly accelerate, making the pace of progress harder to track.
  • Page 291 What alternatives are available? The more that safe alternatives are available, the easier it is to phase out use.
  • Page 291 Does it have autonomous characteristics?
  • Page 291 The more a technology by design requires human intervention, the less chance there is of losing control.
  • Page 292 Orienting development toward defense over offense tends toward containment.
  • Page 292 The talent available for a synthetic biology start-up is, in global terms, still quite small. Both help containment in the near term.
  • Page 292 Specific technologies are easier to regulate than omni-use technologies, but regulating omni-use is more important.
  • Page 292 If you can keep price and ease of access out of reach for many, proliferation becomes more difficult.
  • Page 293 The reality is, we have often not controlled or contained technologies in the past. And if we want to do so now, it would take something dramatically new, an all-encompassing program of safety, ethics, regulation, and control that doesn't even really have a name and doesn't seem possible in the first place.
  • Page 293 A useful comparison here is climate change. It too deals with risks that are often diffuse, uncertain, temporally distant, happening elsewhere, lacking the salience, adrenaline, and immediacy of an ambush on the savanna—the kind of risk we are well primed to respond to. Psychologically, none of this feels present. Our prehistoric brains are generally hopeless at dealing with amorphous threats like these.
  • Page 294 Pessimism aversion is much harder when the effects are so nakedly quantifiable.
  • Page 294 There's no handy metric of risk, no objective unit of threat shared in national capitals, boardrooms, and public sentiment, no parts per million for measuring what technology might do or where it is.
  • Page 294 No popular movement behind stopping it, no graphic images of melting icebergs and stranded polar bears or flooded villages to raise awareness.
  • Page 294 The first step is recognition.
  • Page 295 The more it's on the public's radar, the better.
  • Page 295 My intent is to seed ideas in the hopes of taking the crucialfirst steps toward containment.

    Chapter 14: Ten Steps Toward Containment

  • In 2023 it's now clear that, compared with the early systems, it is extremely difficult to goad something like ChatGPT into racist comments. Is it a solved problem? Absolutely not.
  • Page 298 A key driver behind this progress is called reinforcement learning from human feedback.
  • Page 300 The number of AI safety researchers is still minuscule:
  • Page 300 Only a handful of institutions, owing to the challenges of resources, take technical safety issues seriously. And yet safety decisions made today will alter
  • Page 301 There's a clear must-do here: encourage, incentivize, and directly fund much more work in this area.
  • Page 302 In AI, technical safety also means sandboxes and secure simulations to create provably secure air gaps so that advanced AIs can be rigorously tested before they are given access to the real world.
  • Page 302 As a user, it's all too easy to be lulled into a false sense of security and assume anything coming out of the system is true.
  • Page 302 Here it's about making sure AI outputs provide citations, sources, and interrogable evidence that
  • Page 302 user can further investigate when a dubious claim arises.
  • Page 303 How can you build secure values into a powerful AI system potentially capable of overriding its own instructions?
  • Page 303 Another ongoing question is how to crack the problem of "corrigibility," ensuring that it is always possible to access and correct systems.
  • Page 303 We should also build robust technical constraints into the development and production process.
  • Page 303 printers are built with technology preventing you from copying or printing money,
  • Page 304 AI systems could be built with cryptographic protections
  • Page 304 building a bulletproof off switch,
  • Page 304 Safety features should not be afterthoughts but inherent design properties of all these new technologies,
  • Page 305 having meaningful oversight and enforceable rules and reviewing technical implementations are vital.
  • Page 305 Trust comes from transparency. We absolutely need to be able to verify, at every level, the safety, integrity, or uncompromised nature of a system.
  • Page 306 "red teaming"—that is, proactively hunting for flaws in AI models or software systems.
  • Page 306 The more this is done publicly and collectively, the better, enabling all developers to learn from one another.
  • Page 306 It's also time to create government-funded red teams that would rigorously attack and stress test every system,
  • Page 306 Systems implemented to keep track of new technologies need to recognize anomalies, unforeseen jumps in capability, hidden failure modes.
  • Page 311 Buying time in an era of hyper-evolution is invaluable. Time to develop further containment strategies.
  • Page 311 Right now technology is driven by the power of incentives rather than the pace of containment.
  • Page 312 Chips aren't the only choke point. Industrial-scale cloud computing, too, is dominated by six major companies.
  • Page 312 So, as negative impacts become clear, we must use these choke points to create sensible rate-limiting factors, checks on the speed of development,
  • Page 314 Credible critics must be practitioners.
  • Page 316 Profit drives the coming wave. There's no pathway to safety that doesn't recognize and grapple with this fact.
  • Page 316 we must find new accountable and inclusive commercial models that incentivize safety and profit alike.
  • Page 316 reconcile profit and social purpose
  • Page 321 Containment needs a new generation of corporations.
  • Page 321 Technological problems require technological solutions, as we've seen, but alone they are never sufficient. We also need the state to flourish. Highlight (blue) - Page 321 The physicist Richard Feynman famously said, "What I cannot create, I do not understand."
  • Page 323 Bodies close to executive power, like the White House's Office of Science and Technology Policy, are growing more influential. More is still needed:
  • Page 323 In 2022 the White House released a blueprint for an AI Bill of Rights with five core principles "to help guide the design, development, and deployment of artificial intelligence and other automated systems so that they protect the rights of the American public." Citizens should, it says, be protected from unsafe and ineffective systems and algorithmic bias. No one should be forced to subject themselves to AI. Everyone has the right to say no. Efforts like this should be widely supported and quickly implemented.
  • Page 324 The most sophisticated AI systems or synthesizers or quantum computers should be produced only by responsible certified developers. As part of their license, they would need to subscribe to clear, binding security and safety standards, following rules, running risk assessments, keeping records, closely monitoring live deployments.
  • Page 324 Different licensing regimes could apply according to model size or capability: the bigger and more capable the model, the more stringent the licensing requirements.
  • Page 325 "tax on robots";
  • Page 325 Tax credits topping up the lowest incomes could be an immediate buffer in the face of stagnating or even collapsing incomes.
  • Page 326 Who is able to design, develop, and deploy technologies like this is ultimately a matter for governments to decide.
  • Page 327 Countries no more like giving up power than companies like missing out on profit, and yet these are precedents to learn from, shards of hope in a landscape riven with resurgent techno-competition.
  • Page 333 what's needed for the coming wave: real, gut-level buy-in from everyone involved in frontier technologies.
  • Page 333 sound bite. For a start, being utterly open about failures even on uncomfortable topics should be met with praise, not insults.
  • Page 334 Researchers must be encouraged to step back from the constant rush toward publication. Knowledge is a public good, but it should no longer be the default.
  • Page 334 In AI, capabilities like recursive self-improvement and autonomy are, I think, boundaries we should not cross.
  • Page 338 Just a few days after the release of GPT-4, thousands of AI scientists signed an open letter calling for a six-month moratorium on researching the most powerful AI models.
  • Page 343 precarious. Safe, contained technology is, like liberal democracy, not a final end state; rather, it is an ongoing process, a delicate equilibrium that must be actively maintained, constantly fought for and protected.
  • Page 343 Some level of policing the internet, DNA synthesizers, AGI research programs, and so on is going to be essential.
  • Page 344 Technologists and the general public alike will have to accept greater levels of oversight and regulation than have ever been the case before.
  • Page 345 And while the sheer scale of the challenge is huge, each section here drills down into plenty of smaller areas where any individual can still make a difference. It will require an awesome effort to fundamentally change our societies, our human instincts, and the patterns of history. It's far from certain. It looks impossible.
  • Page 345 meeting the great dilemma of the twenty-first century must be possible. Life After the Anthropocene
  • Page 350 the same industrial technologies that caused so much pain gave rise to a prodigious improvement in living standards.
  • Page 350 Decades, centuries later, the descendants of those weavers lived in conditions the Luddites could have scarcely imagined,
  • Page 350 The coming wave is going to change the world. Ultimately, human beings may no longer be the primary planetary drivers, as we have become accustomed to being.
  • Page 351 The Luddite reaction is natural, expected. But as always, it will be futile.