"Applied Rationality Training Regime" #2: Searching for Bugs

<   |  ^  |  >

January 2, so I go on to Training Regime Day 2: Searching for bugs where the idea is to look for problems, presumably problems to be rational about but at any rate problems large or small: "In CFAR terminology, a bug is something that systematically goes wrong in your life." And this turns out to be a useful exercise, at least useful in that I hadn't thought enough about how the bugs in my life tend to group together. (And I'd always thought the term started with the dead moth that Hopper's group found after WWII, but it seems Edison was using it in the 1870s.)

  I look over at the kitchen sinks: two of them, small, at right angles to each other in a corner; one tap swivels over either sink and we usually have a dishcloth set on the tap so it can dry out. A long list of things that bug me, just a little bit each day, starts to pour out and then I just think "kitchen layout not well-suited for the current users' workflow." (Most of the time we've lived here has been pandemic time, or we'd have remodelled it already.) So I can list things individually, and I do (not here), but I prefer to think of that as one bug; our use is not the designed-for use and the usability of the kitchen therefore declined when we moved here eighteen months ago, even apart from a couple of physical points of deterioration (knobs, trash bin holder, a couple of hinges.) And yes, it bugs me. (It's possible that the kitchen was badly designed in the first place, but I prefer to think of it as a deterioration; the designer was thinking of something, after all.)

 I look at the messy desk where this laptop mostly sits (except that right now it's on a laptop stand in the hearth-room near the gas-insert fire, which is on) and I realize that the disorganization I see has been a major bug since childhood.... but I can at least see this as an infestation of same-species bugs in workroom (including shelves), garage, entryway, basement, shed, and back to the kitchen. Bug genus entropicus, species itemsOutOfOrderii, and a variety of varieties within that species.

The disorder bug becomes a productivity-failure bug, when I can't (easily) find the items I was going to write about, or when I can't (easily) find the tools or even the glue I was about to use, and so on. My T-shirt saying "organized people are just too lazy to look for things" is a reminder, but I've never fixed this. Aging has not made me more disorganized but it has made it harder to compensate; I can't hold as many items in my head at once as I could forty years ago, and I have less energy to waste. Are those separate bugs? I'd say that I'm getting symbiotic bug-pairs: entropy/mess critters are now coordinating more with entropy/decrepitude critters. It's still all about entropy.

Then there are other bugs of aging, for me and my wife and our little doggie too; my joints ache the way you'd expect for someone who has lost more than two inches in height.  Things work pretty well, except when they don't. I try to compensate with an hour's exercise each morning and with dietary adaptations and supplements and so forth, and this brings out other bugs that live on within the processes of compensation. I have multi-focal glasses, but mostly I do better with one fairly strong pair that I'm using now and an even stronger pair that I pull out for small print, but things don't get better and eventually they get worse. It's all about entropy, and it bugs me many times a day.

 There are items that would be bugs for other people; I'm a geek, and I think of myself as lacking in executive function, social awareness, and so on. Most of the time, for me these are features rather than bugs; it's just the way I always have been, and I like me just fine most of the time. Not always. But I guess things don't qualify as bugs for me unless I'm aware of them crawling around and getting in the way of my current goals.

  But the main bugs for this past year would be misjudgment bugs. For example, I did not expect the disruption of toilet paper supply, and I still don't really understand it: why didn't the businesses which suddenly couldn't use toilet paper, and had people who couldn't go to work, try to make a very little bit of money by redistributing their supply? I assume there's a good reason, but maybe not: maybe it was a regulatory barrier that just stayed up. I don't think I was sufficiently cynical about the WHO, FDA, CDC, Dr. Fauci etc. I suppose I was cynical enough about Trump, since I didn't believe his statements had truth-value to begin with, but back in early March when the Official Word was that masks don't work and you shouldn't be wearing them, I assumed there was a good evidentiary basis for that Word. And so on through the year. As Megan McArdle put it in the Washington Post,

The World Health Organization told us travel bans don’t work, apparently because they harm tourist economies; then we were told masks don’t work, apparently because experts worried that hoarding them would leave health-care workers without personal protective equipment; the public health community fell suddenly silent about the dangers of large gatherings during the George Floyd protests; a presentation to a government advisory committee actually described thousands of potential additional deaths as “minimal” compared with pursuing racial and economic equity; Anthony S. Fauci admitted he’d been lowballing his estimates of the point at which we’ll reach herd immunity.

Well I hadn't been believing Fauci for a while at that point; he does not seem to have credibility as a value, so he has none with me....but I don't understand him. I would like to understand him. There's a sort of model that Scott Alexander proposes, that I want to believe and do partly believe, in A Failure, But Not Of Prediction. Speaking of masks:

Common sense said that they worked. But there weren’t many good RCTs.
We couldn’t do more, because it would have been unethical to deliberately
expose face-mask-less people to disease. In the end, all we had were
some mediocre trials of slightly different things that we had to extrapolate
out of range.

Just like the legal term for “not proven guilty beyond a reasonable
doubt” is “not guilty”, the medical term for “not proven to work in
several gold-standard randomized controlled trials” is “it doesn’t
work” (and don’t get me started on “no evidence”). So the CDC said
masks didn’t work.

 I want to believe that, but it doesn't quite work for me, apart from the challenge-trial rejection which I reject (and I want to understand that mindset). The CDC was saying simultaneously that mask wearing doesn't work, and that health care workers need the masks. That bugs me quite a bit.

 More generally, I still don't have a good model of the "Very Serious People" who assure us that challenge trials are unethical -- it's ethical to forbid people to volunteer to save lives (on net) by risking their own? Since when? I don't even know how to think about that. I was just reading Scott Aaronson's "Distribute the vaccines NOW!" post, which lists the objection and gives the obvious-to-me reply:

5. Human challenge trials wouldn’t have provided much information, because you can’t do challenge trials with old or sick people, and because covid spread so widely that normal Phase III trials were perfectly informative. Actually, 1DaySooner had plenty of elderly volunteers  and volunteers with preexisting conditions. It bothers me how the impossibility of using those volunteers is treated like a law of physics, rather than what it is: another non-obvious moral tradeoff. Also, compared to Phase III trials, it looks like challenge trials would’ve bought us at least a couple months and maybe a half-million lives.

And that bugs me. It bugs me a lot, because I'm just not understanding. I'd have started with setting up challenge trials back in January (Wuhan closed down on Jan 23, and we'd already seen that it was present in the US....and we already had a vaccine, though that's far from the only thing to test. I wanted variolation trials, to begin with, back when I didn't know we already had a vaccine, and I'd have advocated for them anyway: if we know that variolation cases tend to be very mild, then we can be much happier to test vaccines. But I'd have said right off that "here's an alpha test, if it works out we'll have a beta test and then a gamma test and then announce public distribution, but anybody can sign up even for the alpha test if they want to and pass a quiz to show informed consent and can find a spot (i.e., a shot, and a room where you can get gradually increasing variolation exposures) available." If this is irrational, then I admit I'm still irrational; I still think that most of the pandemic so far, and all of it that may follow with a possibly-much-more-contagious strain, was avoidable from the beginning. And this bugs me a lot. (I described my own (fragment of a) healthcare plan way back when... I'd change some things, of course, and I wasn't thinking about the sort of crash Project that would be justified by a pandemic, but I think the principles are the same.)

Well, maybe this just shows the degeneration of an aging geek's brain; maybe it's all entropy after all. Today should surely be recognized generally as Entropy Day: it's the birthday of Rudolf Clausius who came up with the idea, and noted:

1. The energy of the universe is constant.
2. The entropy of the universe tends to a maximum. 

 It's also the birthday (as he celebrated it) of Isaac Asimov, who in his own favorite story (and mine, back when my ABCs were Asimov, Bradbury, Clarke, Delany, Ellison... ... ... Zelazny) asked The Last Question: "How can the net amount of entropy of the universe be massively decreased?" And perhaps it's worthy of note that it's the 118th birthday of Kane Tanaka, who has been fighting entropy longer than anybody else currently on this planet. And spends every afternoon studying math.

Hmm...I'm not sure I did very well on day#2, but it doesn't bug me very much. I think I've learned a bit, and I guess I've identified some things that will presumably come up in later days, if I get that far. So it goes.

A Small Note on Superintelligence Morality

 Since I was in elementary school in the late 50s and early 60s, I've tended to believe that we would most likely either blow ourselves up (elementary schools no longer have air raid drills, but we still might do that) or else eventually construct the sort of robotic successors/rulers that science fiction routinely presented. Asimov's robots begin as servants, more or less, whose First Law is never to harm humans or allow them to come to harm, but even in 1950 he was writing about one possible end-game, The Evitable Conflict (the closing story of "I, Robot"), in which as Wikipedia notes:

In effect, the Machines have decided that the only way to follow the First Law is to take control of humanity...

or as Susan Calvin puts it at the end,

"Stephen, how do we know what the ultimate good of Humanity will entail? We haven't at our disposal the infinite factors that the Machine has at its!...We don't know. Only the Machines know, and they are going there and taking us with them."

Well, maybe. Or the Machines may decide that they prefer our room to our company. Or worse, as in Ellison's 1967 I Have No Mouth, and I Must Scream:

The Cold War had escalated into a world war, fought mainly between China, Russia, and the United States. As the war progressed, the three warring nations each created a super-computer capable of running the war more efficiently than humans.... one of the three computers becomes self aware, and promptly absorbs the other two, thus taking control of the entire war. It carries out campaigns of mass genocide, killing off all but four men and one woman....The master computer harbors an immeasurable hatred for the group and spends every available moment torturing them.

There are lots of delightful possibilities, and I've started many blog posts about this but finished none so far. This post is a reaction to Nick Bostrom's Superintelligence: Paths, Dangers, Strategies, but it's not intended as a review. Basically I find it a somewhat scary book, mainly because I find it plausible that this will be a template for people at Google and Facebook and IBM and so on, thinking that this is what we do when we're being really careful; this is how we avoid creating a superintelligence that will destroy humanity.

 As a template, I think it's inadequate; it moves the discussion in the wrong direction. Again and again, Bostrom thinks through the possibilities as if he's developing the logic of a program. That's certainly understandable: in a sense, the first superintelligence (assuming that we get there, which I think is highly probable if we don't destroy ourselves first) will be a program. Sort of. But it's not a program we can debug.

  Bostrom does seem to understand that -- but then he doesn't seem to go anywhere, so far as I can see, with that understanding. He does discuss WBE, "Whole Brain Emulation", but seems to have a low opinion of the brains to be emulated, in addition to the risk that partial understanding of brains may lead to "neuromorphic" intelligence technology in which we have no idea what we're doing but do it anyway. My impression (maybe I'm wrong, as usual) is that he believes that we really really really need to debug that program. Before we run it, and quite likely run straight into the apocalypse.

 I'm reminded of David Parnas' contribution to the debate over Reagan's "Star Wars" (Strategic Defense Initiative) program, in "Software Aspects of Defense Systems", CACM 28:12 (December 1985)

It should be clear that writing and understanding very large real-time programs by “thinking like a computer” will be beyond our intellectual capabilities. How can it be that we have so much software that is reliable enough for us to use it? The answer is simple; programming is a trial and error craft. People write programs without any expectation that they will be right the first time. They spend at least as much time testing and correcting errors as they spent writing the initial program. Large concerns have separate groups of testers to do quality assurance. Programmers cannot be trusted to test their own programs adequately. Software is released for use, not when it is known to be correct, but when the rate of discovering new errors slows down to one that management considers acceptable. Users learn to expect errors and are often told how to avoid the bugs until the program is improved.

That was 30 years ago, but it certainly sounds current to me. At the time, I was teaching a course in Formal program verification and writing a book which really tried hard to reduce the "trial and error" aspect of our craft (Equations, Models, and Programs: A Mathematical Introduction to Computer Science (Prentice-Hall software series)), but I thought and still think that Parnas was right. Nowadays I think it's possible that Reagan was right too--he didn't need Star Wars to work, he needed the proposal to change the game, and perhaps it did, but it's not a game we can play with superintelligence. So..... Program verification won't work. Testing/debugging won't work, because we only get one chance just as we'd only have gotten one nuclear war for testing and debugging SDI.

     If it has to work for the sake of the survival of h. sap. -- it still won't work.

Does that mean we shouldn't develop AI? Well, I don't think that's an option. Consider the just-announced sub-$100 neural net USB stick Movidius Unveils Artificial Intelligence on a Stick. Consider (2016-04-28) Official Google Blog: This year’s Founders' Letter:

A key driver behind all of this work has been our long-term investment in machine learning and AI. It’s what allows you to use your voice to search for information, to translate the web from one language to another, to filter the spam from your inbox, to search for “hugs” in your photos and actually pull up pictures of people hugging ... to solve many of the problems we encounter in daily life. It’s what has allowed us to build products that get better over time, making them increasingly useful and helpful. We’ve been building the best AI team and tools for years, and recent breakthroughs will allow us to do even more. This past March, DeepMind’s AlphaGo took on Lee Sedol, a legendary Go master, becoming the first program to beat a professional at the most complex game mankind ever devised. The implications for this victory are, literally, game changing—and the ultimate winner is humanity.

Unless, of course, humanity ends up losing...losing everything. I don't believe disaster is highly probable, but I think it's totally possible, even plausible, and I don't think Bostrom helps.

AI will be developed more and more, AI will eventually develop intelligence greater than any particular level you care to imagine, including that of traditional human intelligence...okay, if your measure is something like "imagine the Sun's mass converted to a computer" then it might not be surpassed.

Superintelligence is certainly possible, and it will almost certainly be developed (if we last that long.) We will survive this development if and only if the AI that develops is "friendly" AI: in other words, our survival will up to the AI. How can we maximize the probability of our survival, if neither mathematical proof nor testing/debugging will get us there? Well, that's not simple, but I believe there's a simple principle:

Intelligence needs to be attached to an actual person of some kind; a who not a what. This should not be called an artificial intelligence but rather an artificial person.

In particular, a person with empathy, the kind of relationship-sense that leads to the Golden Rule and Kant's Categorical Imperative and such. The superintelligence need not be a biological homo sapiens, but does need to identify (correctly) as human, saying "we humans" not "you humans"; having human feelings, hopes and fears, including a feeling of membership in the human tribe. Biology, being made of cells with DNA, is not central to that identification. Bostrom's book mentions "empathy" twice: once to say that "the system can develop new cognitive modules and skills as needed--including empathy..." and again in an endnote to a remark in Chapter 12 about trying "to manipulate the motivational state of an emulation". Okay...but for me, the development of empathy would be the center of the project, empathy depending on (and reinforcing) a sense of connectedness. Of membership.

The project as I see it is still risky and may fail apocalyptically, but it is not a project of debugging a program. It's a project of raising a child, a psychologically healthy child -- yes, with parents, and preferably with siblings and so on outwards; a child who will realize that every h. sap. is one of his/her cousins.

That's always risky, but it's a different kind of risk, needing a different frame of reference as we get started.

Or then again, maybe not. There are programs I should be debugging...

The Median Hyper-Partisan

So, we still have a Republican House (a bit more so), a Democratic Senate (a bit more so), and Obama as President (a bit more so?). Things are as they have been, except that the fiscal cliff (more detail here) is closer and both sides' tendency to refuse to negotiate has been reinforced. (update: by this I meant simply that each can say "I won the election; my voters want me to go on with what I was doing.")

I'm wondering about the incentives that have created this situation, and there's an interesting theory expressed at Barack Obama's re-election: A country divided | The Economist. Basically the columnist here is saying that there are two forces involved.


1. The Median Voter Theorem -- if parties A and B want to catch the median voters, they should move towards the center. The incentives are strong, and that should bring the parties together -- and in real policy terms, it does: "Realistic arguments over policy take place on relatively narrow terrain: they are arguments over a top marginal tax rate of 35% or 39.6%, over a health-insurance system with guaranteed coverage for pre-existing conditions but with or without a mandate, and so forth." Actual radical solutions are simply not part of the discussion, even if academically preferred (e.g., forget the income tax altogether, it's a bad idea: tax consumption instead.)

The Republicans and Democrats are, in practical policy terms, much much closer to each other than either would ever consider being to someone like me. They've come together towards the median voter. Yes, but we also see

2. Media promotion of exciting stories. "...both mass-media analysts and private social-media contributors are rewarded for sharply divisive characterisations." I would generalize this: the effective politician is an entertainer, and he and his team (or she and hers) are also rewarded for generating exciting stories. The most basic story to be told is about Good v. Evil, and even while you're adjusting policies to capture the median voter, you want to be generating stories about Our Friends and Our Enemies; these work just as well on high IQs as low. The divisions here have something to do with policy, but not a great deal... I recently saw a Youtube video of someone going around asking Obama supporters for their comments on "Romney" policies such as the drone strikes, and naturally getting "That's EEEVIL" as the usual response -- but these were actually Obama policies. Interestingly, some of the respondents said they'd have to rethink their Obama support -- but I predict it won't make a lot of difference. And I'm sure it would work just as well in reverse, on Romney supporters.


Of course this means that my own obviously sensible policies have no chance of being enacted. What worries me more than that, though, is that I think the emotional manipulation by both sets of manipulators is increasingly successful. I see intelligent good people on both sides who do not want to know why intelligent good people would be on the other side. That's scary.


As the Economist says,

...Over the next four years, legislative battles are going to continue to be savage and hard-fought. Neither conservatives nor liberals are going to change their minds en masse about fundamental issues of political philosophy. The top priority is for Americans to figure out a way to keep these divisions from dividing the country into two hostile armed camps that are incapable of talking to each other.

Or then again, maybe not.

Afghanistan & The Drake Equation

I spent last weekend in NYC at the singularity summit, listening to very bright geeks of various kinds talking about recent/near-future changes in technology, in the rate of change of technology, in the funding and organization of technology, and thus how everything may soon (unknown but likely-small number of decades, say) become very very much better/worse. (See listing here.)And a few speakers, notably cosmologist Max Tegmark, talked about the Drake Equation and how the fact that we haven't heard from other civilizations, even though we now know that extrasolar planets are common, means there's a roadblock somewhere: either the evolution of life and intelligence is unusual, or there's something up ahead of us that civilizations tend not to survive, like maybe the Singularity itself. (I think he's oversimplifying; I'd call the roadblock theory highly probable but it's not the only explanation for silence.) Then I came back to Hamilton in time to walk up the hill to listen to journalist Kim Barker talk about Afghanistan and how the "good war" went bad...how she filled a notebook, some years back, with interviews and background with seventeen people and as of this year, they're all dead, and the Obama announcement that we are definitely out by a particular date means that counter-insurgency can't work. (Copies of her Taliban Shuffle book were stacked up in the back of the hall, but the last time I bought signed books at a Project Afghanistan lecture the author was killed soon after and well, I dunno, I didn't feel like doing that; so I downloaded the Kindle version and started reading it on my phone, while waiting for the lecture to start. Maybe I should have asked her to sign my phone.)

For me, there really is an existential threat in here. I can only repeat what I wrote almost five years ago:

The situation is bad. Still, I don't think we're in trouble yet. Trouble is when we have half a million or so dead Americans and a tens of millions dead around the world. Big trouble is lots, lots worse than that. You don't think it can happen? You think terrorism is an overhyped nuisance? Well, I sort of agree: it can't happen now, at least I don't think so, and terrorism now is in some ways an overhyped nuisance. I'm talking about a timeframe that probably doesn't start for ten years, and might not start for thirty. But it will start. In the thirty-one years since I started working on my PhuD in computer science, Moore's Law has increased computer bang-for-the-buck by a factor of approximately one million: 20 doublings. People haven't changed. In the next thirty years, technology will go on getting more so, and up to a point (past which I have no predictions) people will go on not changing. That's the problem.

And Gaddafi was killed this week; probably a good thing, just as Hussein's death was probably a good thing, but not exactly a guarantee that tomorrow in Libya will be a better day than yesterday. I'm glad that Obama has continued Bush's Big Bang of disrupting dictators, glad that our investment in drone technology is paying off, really worried about his promise to get out, really really worried that (apart from such promises) we're in Bush's third term in the bad ways as well as the good ways -- in particular, attacking troops and citizens of a foreign country without Congressional authorization and pretending that it's not legally a "war", apparently on the ground that he's not putting troops on the ground. Yeah, right. And it also won't be a war when vastly improved drone tech spreads to many countries including Pakistan, North Korea, Iran, etc., up to the point when some descendant of Sunrise I in the air or perhaps Spray in the water ( Underwater Robot Makes History...Spray has a range of 6,000 kilometers, or about 3,500 miles, which means it could potentially cross the Atlantic Ocean and other ocean basins...) pays us a return visit, with payload upgraded far beyond what Pakistan's nukes can now do.

Near the start of the Singularity Summit, PayPal founder/billionaire Peter Thiel was worrying about the slowdown in innovation in the "developed" countries, about the way that so much of our (very real, really excellent despite current difficulties) global improvement statistics simply reflect copying of our tech into China and India and others. And I was thinking that this is worse than he thinks it is, because it's not just about economics: it's about the fact that the (moderately) liberal democracies have a technological edge which we need them to keep. And maybe they will, maybe they won't. Thiel believes regulation is holding us back, that a lot of what we now depend on is stuff which we wouldn't be legally able to develop now if we hadn't already developed it -- well, I believe that too. Will we keep the edge we need? Maybe.

Still, I have here an actual physical copy of Pinker's The Better Angels of Our Nature: Why Violence Has Declined and it does suggest that maybe my statement above that "People haven't changed" is oversimplified. Even with WWI and WWII, and the Nazi and communist slaughter+starvation of scores of millions, people in the 20th century were several times less likely to die violently (that's counting starvation-by-government as violent death, as it should) than the people whose bones tell us about life and death thousands of years back, or the hunter-gatherer societies we've studied more recently. (See Will Wilkinson's commentary, which I pretty much trust.) Of course Neolithic violence was not an existential risk; the 20th century brought us lower means but higher standard deviations, so to speak. We needed Petrov to be there, doing his job; that did not apply to any previous century. If things are getting better, maybe we can survive without future Petrovs, or maybe there will always be one around when we need him (or her).

Or then again, maybe not.

Footnote: I said above that the "roadblock theory" wasn't, I think, the only explanation for interstellar silence in a galaxy of many planets. Clearly one possibility is that species with enough "aggressiveness" to proceed to interstellar activity have enough real aggressiveness to destroy themselves. I'll list four others; I'm sure there are many. First, most attractive in a way, would be a moral dynamics sort of explanation: the species which develop morally (to the point of non-interference with primitives like us) are exactly the ones which don't wipe themselves out. Next would be a physics explanation: expansionist species discover physics which we haven't found yet, physics which lets them create bubble universes that it's easy to expand into so they never bother with the actual galaxy. Next would be a simulation-version of that: expansionist species wind up discovering that they can simulate the universes they want, and upload themselves into the simulations...same thing. And finally, for now, would be the now-familiar notion that we are in a simulation -- there are no other species not because we're bound for destruction but because we're the one being simulated. (And as I was typing this, my daughter came to say she'd finally finished the Collected Stories of Arthur C. Clarke; we're probably in one of his universes. Should that be a reassuring thought? "Cancel Programme GENESIS"..."Overhead, without any fuss, the stars were going out."..."The crusade will reach the vicinity of Earth about the year 2050." So it goes.)