A Small Note on Superintelligence Morality

 Since I was in elementary school in the late 50s and early 60s, I've tended to believe that we would most likely either blow ourselves up (elementary schools no longer have air raid drills, but we still might do that) or else eventually construct the sort of robotic successors/rulers that science fiction routinely presented. Asimov's robots begin as servants, more or less, whose First Law is never to harm humans or allow them to come to harm, but even in 1950 he was writing about one possible end-game, The Evitable Conflict (the closing story of "I, Robot"), in which as Wikipedia notes:

In effect, the Machines have decided that the only way to follow the First Law is to take control of humanity...

or as Susan Calvin puts it at the end,

"Stephen, how do we know what the ultimate good of Humanity will entail? We haven't at our disposal the infinite factors that the Machine has at its!...We don't know. Only the Machines know, and they are going there and taking us with them."

Well, maybe. Or the Machines may decide that they prefer our room to our company. Or worse, as in Ellison's 1967 I Have No Mouth, and I Must Scream:

The Cold War had escalated into a world war, fought mainly between China, Russia, and the United States. As the war progressed, the three warring nations each created a super-computer capable of running the war more efficiently than humans.... one of the three computers becomes self aware, and promptly absorbs the other two, thus taking control of the entire war. It carries out campaigns of mass genocide, killing off all but four men and one woman....The master computer harbors an immeasurable hatred for the group and spends every available moment torturing them.

There are lots of delightful possibilities, and I've started many blog posts about this but finished none so far. This post is a reaction to Nick Bostrom's Superintelligence: Paths, Dangers, Strategies, but it's not intended as a review. Basically I find it a somewhat scary book, mainly because I find it plausible that this will be a template for people at Google and Facebook and IBM and so on, thinking that this is what we do when we're being really careful; this is how we avoid creating a superintelligence that will destroy humanity.

 As a template, I think it's inadequate; it moves the discussion in the wrong direction. Again and again, Bostrom thinks through the possibilities as if he's developing the logic of a program. That's certainly understandable: in a sense, the first superintelligence (assuming that we get there, which I think is highly probable if we don't destroy ourselves first) will be a program. Sort of. But it's not a program we can debug.

  Bostrom does seem to understand that -- but then he doesn't seem to go anywhere, so far as I can see, with that understanding. He does discuss WBE, "Whole Brain Emulation", but seems to have a low opinion of the brains to be emulated, in addition to the risk that partial understanding of brains may lead to "neuromorphic" intelligence technology in which we have no idea what we're doing but do it anyway. My impression (maybe I'm wrong, as usual) is that he believes that we really really really need to debug that program. Before we run it, and quite likely run straight into the apocalypse.

 I'm reminded of David Parnas' contribution to the debate over Reagan's "Star Wars" (Strategic Defense Initiative) program, in "Software Aspects of Defense Systems", CACM 28:12 (December 1985)

It should be clear that writing and understanding very large real-time programs by “thinking like a computer” will be beyond our intellectual capabilities. How can it be that we have so much software that is reliable enough for us to use it? The answer is simple; programming is a trial and error craft. People write programs without any expectation that they will be right the first time. They spend at least as much time testing and correcting errors as they spent writing the initial program. Large concerns have separate groups of testers to do quality assurance. Programmers cannot be trusted to test their own programs adequately. Software is released for use, not when it is known to be correct, but when the rate of discovering new errors slows down to one that management considers acceptable. Users learn to expect errors and are often told how to avoid the bugs until the program is improved.

That was 30 years ago, but it certainly sounds current to me. At the time, I was teaching a course in Formal program verification and writing a book which really tried hard to reduce the "trial and error" aspect of our craft (Equations, Models, and Programs: A Mathematical Introduction to Computer Science (Prentice-Hall software series)), but I thought and still think that Parnas was right. Nowadays I think it's possible that Reagan was right too--he didn't need Star Wars to work, he needed the proposal to change the game, and perhaps it did, but it's not a game we can play with superintelligence. So..... Program verification won't work. Testing/debugging won't work, because we only get one chance just as we'd only have gotten one nuclear war for testing and debugging SDI.

     If it has to work for the sake of the survival of h. sap. -- it still won't work.

Does that mean we shouldn't develop AI? Well, I don't think that's an option. Consider the just-announced sub-$100 neural net USB stick Movidius Unveils Artificial Intelligence on a Stick. Consider (2016-04-28) Official Google Blog: This year’s Founders' Letter:

A key driver behind all of this work has been our long-term investment in machine learning and AI. It’s what allows you to use your voice to search for information, to translate the web from one language to another, to filter the spam from your inbox, to search for “hugs” in your photos and actually pull up pictures of people hugging ... to solve many of the problems we encounter in daily life. It’s what has allowed us to build products that get better over time, making them increasingly useful and helpful. We’ve been building the best AI team and tools for years, and recent breakthroughs will allow us to do even more. This past March, DeepMind’s AlphaGo took on Lee Sedol, a legendary Go master, becoming the first program to beat a professional at the most complex game mankind ever devised. The implications for this victory are, literally, game changing—and the ultimate winner is humanity.

Unless, of course, humanity ends up losing...losing everything. I don't believe disaster is highly probable, but I think it's totally possible, even plausible, and I don't think Bostrom helps.

AI will be developed more and more, AI will eventually develop intelligence greater than any particular level you care to imagine, including that of traditional human intelligence...okay, if your measure is something like "imagine the Sun's mass converted to a computer" then it might not be surpassed.

Superintelligence is certainly possible, and it will almost certainly be developed (if we last that long.) We will survive this development if and only if the AI that develops is "friendly" AI: in other words, our survival will up to the AI. How can we maximize the probability of our survival, if neither mathematical proof nor testing/debugging will get us there? Well, that's not simple, but I believe there's a simple principle:

Intelligence needs to be attached to an actual person of some kind; a who not a what. This should not be called an artificial intelligence but rather an artificial person.

In particular, a person with empathy, the kind of relationship-sense that leads to the Golden Rule and Kant's Categorical Imperative and such. The superintelligence need not be a biological homo sapiens, but does need to identify (correctly) as human, saying "we humans" not "you humans"; having human feelings, hopes and fears, including a feeling of membership in the human tribe. Biology, being made of cells with DNA, is not central to that identification. Bostrom's book mentions "empathy" twice: once to say that "the system can develop new cognitive modules and skills as needed--including empathy..." and again in an endnote to a remark in Chapter 12 about trying "to manipulate the motivational state of an emulation". Okay...but for me, the development of empathy would be the center of the project, empathy depending on (and reinforcing) a sense of connectedness. Of membership.

The project as I see it is still risky and may fail apocalyptically, but it is not a project of debugging a program. It's a project of raising a child, a psychologically healthy child -- yes, with parents, and preferably with siblings and so on outwards; a child who will realize that every h. sap. is one of his/her cousins.

That's always risky, but it's a different kind of risk, needing a different frame of reference as we get started.

Or then again, maybe not. There are programs I should be debugging...

Afghanistan & The Drake Equation

I spent last weekend in NYC at the singularity summit, listening to very bright geeks of various kinds talking about recent/near-future changes in technology, in the rate of change of technology, in the funding and organization of technology, and thus how everything may soon (unknown but likely-small number of decades, say) become very very much better/worse. (See listing here.)And a few speakers, notably cosmologist Max Tegmark, talked about the Drake Equation and how the fact that we haven't heard from other civilizations, even though we now know that extrasolar planets are common, means there's a roadblock somewhere: either the evolution of life and intelligence is unusual, or there's something up ahead of us that civilizations tend not to survive, like maybe the Singularity itself. (I think he's oversimplifying; I'd call the roadblock theory highly probable but it's not the only explanation for silence.) Then I came back to Hamilton in time to walk up the hill to listen to journalist Kim Barker talk about Afghanistan and how the "good war" went bad...how she filled a notebook, some years back, with interviews and background with seventeen people and as of this year, they're all dead, and the Obama announcement that we are definitely out by a particular date means that counter-insurgency can't work. (Copies of her Taliban Shuffle book were stacked up in the back of the hall, but the last time I bought signed books at a Project Afghanistan lecture the author was killed soon after and well, I dunno, I didn't feel like doing that; so I downloaded the Kindle version and started reading it on my phone, while waiting for the lecture to start. Maybe I should have asked her to sign my phone.)

For me, there really is an existential threat in here. I can only repeat what I wrote almost five years ago:

The situation is bad. Still, I don't think we're in trouble yet. Trouble is when we have half a million or so dead Americans and a tens of millions dead around the world. Big trouble is lots, lots worse than that. You don't think it can happen? You think terrorism is an overhyped nuisance? Well, I sort of agree: it can't happen now, at least I don't think so, and terrorism now is in some ways an overhyped nuisance. I'm talking about a timeframe that probably doesn't start for ten years, and might not start for thirty. But it will start. In the thirty-one years since I started working on my PhuD in computer science, Moore's Law has increased computer bang-for-the-buck by a factor of approximately one million: 20 doublings. People haven't changed. In the next thirty years, technology will go on getting more so, and up to a point (past which I have no predictions) people will go on not changing. That's the problem.

And Gaddafi was killed this week; probably a good thing, just as Hussein's death was probably a good thing, but not exactly a guarantee that tomorrow in Libya will be a better day than yesterday. I'm glad that Obama has continued Bush's Big Bang of disrupting dictators, glad that our investment in drone technology is paying off, really worried about his promise to get out, really really worried that (apart from such promises) we're in Bush's third term in the bad ways as well as the good ways -- in particular, attacking troops and citizens of a foreign country without Congressional authorization and pretending that it's not legally a "war", apparently on the ground that he's not putting troops on the ground. Yeah, right. And it also won't be a war when vastly improved drone tech spreads to many countries including Pakistan, North Korea, Iran, etc., up to the point when some descendant of Sunrise I in the air or perhaps Spray in the water ( Underwater Robot Makes History...Spray has a range of 6,000 kilometers, or about 3,500 miles, which means it could potentially cross the Atlantic Ocean and other ocean basins...) pays us a return visit, with payload upgraded far beyond what Pakistan's nukes can now do.

Near the start of the Singularity Summit, PayPal founder/billionaire Peter Thiel was worrying about the slowdown in innovation in the "developed" countries, about the way that so much of our (very real, really excellent despite current difficulties) global improvement statistics simply reflect copying of our tech into China and India and others. And I was thinking that this is worse than he thinks it is, because it's not just about economics: it's about the fact that the (moderately) liberal democracies have a technological edge which we need them to keep. And maybe they will, maybe they won't. Thiel believes regulation is holding us back, that a lot of what we now depend on is stuff which we wouldn't be legally able to develop now if we hadn't already developed it -- well, I believe that too. Will we keep the edge we need? Maybe.

Still, I have here an actual physical copy of Pinker's The Better Angels of Our Nature: Why Violence Has Declined and it does suggest that maybe my statement above that "People haven't changed" is oversimplified. Even with WWI and WWII, and the Nazi and communist slaughter+starvation of scores of millions, people in the 20th century were several times less likely to die violently (that's counting starvation-by-government as violent death, as it should) than the people whose bones tell us about life and death thousands of years back, or the hunter-gatherer societies we've studied more recently. (See Will Wilkinson's commentary, which I pretty much trust.) Of course Neolithic violence was not an existential risk; the 20th century brought us lower means but higher standard deviations, so to speak. We needed Petrov to be there, doing his job; that did not apply to any previous century. If things are getting better, maybe we can survive without future Petrovs, or maybe there will always be one around when we need him (or her).

Or then again, maybe not.

Footnote: I said above that the "roadblock theory" wasn't, I think, the only explanation for interstellar silence in a galaxy of many planets. Clearly one possibility is that species with enough "aggressiveness" to proceed to interstellar activity have enough real aggressiveness to destroy themselves. I'll list four others; I'm sure there are many. First, most attractive in a way, would be a moral dynamics sort of explanation: the species which develop morally (to the point of non-interference with primitives like us) are exactly the ones which don't wipe themselves out. Next would be a physics explanation: expansionist species discover physics which we haven't found yet, physics which lets them create bubble universes that it's easy to expand into so they never bother with the actual galaxy. Next would be a simulation-version of that: expansionist species wind up discovering that they can simulate the universes they want, and upload themselves into the simulations...same thing. And finally, for now, would be the now-familiar notion that we are in a simulation -- there are no other species not because we're bound for destruction but because we're the one being simulated. (And as I was typing this, my daughter came to say she'd finally finished the Collected Stories of Arthur C. Clarke; we're probably in one of his universes. Should that be a reassuring thought? "Cancel Programme GENESIS"..."Overhead, without any fuss, the stars were going out."..."The crusade will reach the vicinity of Earth about the year 2050." So it goes.)

Paint

I was just talking with Seth Rochford, busy putting paint and polyurethane in our not-quite-complete carriage-house. I mentioned some of my longer-range thoughts about the geo-exchange heating system (which may finally be turned on this week, yay!) and he replied that no matter what happened with that kind of technology, he figured he'd still be using brushes, slapping chemicals on walls. I disagreed.

Start from the fact that the surfaces will all be recorded in 3D modeling programs; I tend to take this for granted. That's easy. So the "painter" will stand with the customer in a virtual space, using graphics better than today's best, choosing colors/textures that will go with each other under various lighting conditions. At the end of this process there will be no more decisions to be made.

We now have a robotic framework which translates the virtual coordinates into physical locations; this can range from a robotic ladder which moves itself along, to a self-driving rented truck which drives to the indicated address and unfolds itself into a framework which covers the building (whether outside or inside). In either case, the framework has tracks. Now look at the inside of your printer, where the print-head rides on rails. (A variety of design options is available here; consider the series-of-2D-printings method for achieving 3D printing, as in Evil Mad Scientist Laboratories' CandyFab: "The X and Y axis motion control systems are based on belt drives and quadrature-encoded motors recycled from two old HP plotters, a large one and a small one".) In any case, for each point in the chosen virtual surface, there is some time-range within which that point can be reached by a robotically wielded "brush". The rest is software.

An alternative to the physical framework is a swarm of mini-robots which crawl or fly over the surface, printing as they go. An even more radical alternative would be mini-(or micro-, or nano-)bots which are the right color/texture, or can change, and which simply go to the right place and form a skin. Suppose that the nano-technology used for this is the nanotech getting most funding right now, namely human DNA: if your paint is made of modified human cells, and if it has a neural component to guide self-repair or calls for replacement, will it have rights?