Trope-a-Day (R): Gone Horribly Right

Gone Horribly Right: An unfortunately large number of experiments with recursively self-improving seed AI are probably the most significant examples, here, for which see that half of the entry under AI Is A Crapshoot.  And for the consequences – well, there are quite the wide variety there, but for a good sampling of the more amusing ones – i.e., the ones that don’t simply get everyone killed immediately – why not pop over to the transhumanist wiki and read your way through the Friendly AI Critical Failure Table:

http://www.acceleratingfuture.com/wiki/Friendly_AI_Critical_Failure_Table

This sort of thing happens on an infrequent yet semi-regular basis.

(The one that happens frequently enough to deserve its own special entry in “civilization-ending stupidities”, though, is when some evangelist-hegemonist religion gets hold of an AI seed and decides to improve its success rate by programming the machine with the Literal Word of God.  Even leaving aside the violations of free will implicit in conversion-by-basilisk-hack, the high probability of a “NOW there is a God!” moment as the growing seed AI tries to reconcile the Literal Word it’s been programmed to believe with the actual universe, and so forth, the end-state in which everyone gets condemned to its lovingly detailed virtual Hell for failure to perfectly comply with the deity’s moral rules in deed and thought every single moment of every day would be a hysterically funny piece of schadenfreude were it not one of the most horrific things ever to happen anywhere.

And yet there’s always some new sucker lined up to set this one off, because clearly they have the real Literal Word of God, so it can’t end in a disaster when they try it…)