According to my cliodynamic studies, it is the case that a policy of routine intervention to prevent any perversion from blooming is suboptimal, inasmuch as it opens up the possibility for bad actors to work around the Coricál Consensus by performing a broad spectrum of experiments in computational theogeny and observing which ones call down intervention teams.
In addition, the perception that DEMIURGE ERRANT will always be there to prevent disasters and clean up the mess weakens the general perception of the field as extremely hazardous to a point which causes a statistically significant increase in the frequency of attempts leading to perversions.
In short: permitting a small number of idiots to have their brains eaten by their errant creations is indeed the best way to prevent a large number of people, mostly lesser idiots, from having their brains eaten by the greater idiots’ errant creations.
Black Box, advisory archai to the Imperial Security Executive
Corícal Consensus
Friendship is Optimal
Thinking briefly of things other than today’s challenge, I’d like to draw the attention of interested readers to the My Little Pony: Friendship is Magic fanfic Friendship is Optimal.
(Trope page here; story here.)
Specifically, with relevance to the Eldraeverse where seed AIs are concerned. Namely, inasmuch as it is a perfect example of what happens when you only screw up the tiniest, most minuscule bit when you had your “Oops, we accidentally a god” moment.
And that’s despite the cosmic horror elements (not counting the wibbling in the comments from people who believe in continuity identity) or the really horrifying implications of a weakly godlike superintelligence that compiles sophonts instrumentally to satisfy the values of other sophonts without sanity-and-ethics checking those values first.
But, hey, most of the human species in this fic gets to continue to exist as minds recognizably descended from their previous iterations and even have their values satisfied. Which, in Eldraeverse terms, means they got absurdly, backyard-moonshot lucky when compared to the set of all people screwing around with computational theogeny. (Especially given the other attempts at seed AI going on in the background.)
And yet.
Which is why the Coricál Consensus is so all-fired important.
(The Transcend, incidentally, would be more than happy to satisfy your values through friendship and ponies, if that’s part of your optimal solution-set. With, y’know, rather tighter consent rules, and ethical constraints, though.)
Doom, Idiocy, and Weirdness
“A few special adhocs aside, the Fifth Directorate is divided into three primary working groups: Existential Threats, Inadvisably Applied Technologies, and Exceptionary Circumstances. Or, as they’re less formally known, the PWGs of Doom, Idiocy, and Weirdness.”
“Existential Threats handles exactly that; the end of everything, or at least everything local. Some of their adhocs are as public as the Fifth ever gets, working on problems like why, exactly, we relative latecomers qualify as one of the eldest of the younger races and why no-one from the Precursor era or earlier seems to be around these days; or preparation for natural disasters like gamma-ray bursts or the upcoming galactic collision. Most of them, though, concentrate on action against more direct threats, like Leviathan Consciousness intrusions, the ambitious that bypassed the Corícal Consensus and incautiously cooked up unstable gods, and any number of insufficiently careful archive-resurrectionists.”
“Inadvisably Applied Technologies is our benevolence PWG. Their adhocs are responsible for intervening in places where we have no particular authority to do so because someone’s playing with fire in the explosives warehouse, and it’s not in anyone’s interest to see a repeat of the Ulijen Disaster. More importantly, it’s especially not in our interest to have people become paranoid about advanced technologies just because someone didn’t read the documentation and flash-fried his entire planet, or worse.”
“Yes, it’s not normally considered appropriate to save people from themselves; but really, that’s just a side-effect of saving large chunks of the rest of the known galaxy from them. Usually, useful ones.”
“Exceptionary Circumstances? We can’t tell you about Exceptionary Circumstances. If we knew what they were or had any idea what to do about them, they wouldn’t be Exceptionary Circumstances. But when we don’t, or we haven’t – that’s what the adhocs of Exceptionary Circumstances do.”
– org briefing to new members of the Select Committee on Imperial State Security
Trope-a-Day: Black Box
Black Box: Quite a few of them lying around in the form of leftover elder race artifacts and other archaeological recoveries. Sensible civilizations and corporations (like Probable Technologies, ICC) really hate this, because they know exactly how Sealed-Evil-In-A-Can dangerous that sort of thing can be, and the likelihood of unknown side effects, and decline to extensively use or commercialize any of them until they’ve figured out not only how to reproduce them, but also just how, exactly, the things work. Very minor, very benign examples may be sold off to collectors, but no-one’s making them a part of their infrastructure until they know all about it.
There are, of course, plenty of sense-challenged people out there.
(On a lesser scale, there are some other examples: the secrets of stabilizing wormholes and building stargates, for example, are both a state secret of the Voniensa Republic and the highest possible grade of commercially-sensitive information for Ring Dynamics, ICC, for reasons in both cases less about maintaining their monopoly and more about wanting to discourage people from screwing with the infrastructure of their really expensive interstellar transportation system – so while the rough details of how they work are known to any schoolchild, that’s about it. Likewise, the algorithms for producing recursively self-improving AI seeds are generally considered proprietary and closely held by informal agreement [the “Corícal Consensus“] of the people who have them, due to the tendency of amateurs to do really stupid things that Go Horribly Right.)
[Of course, in fairness to everyone else, it’s not like in their universe they ever ran into a recovered Black Box that was quite so all-fired useful as, say, Mass Effect‘s mass relay network. On the other hand, I am fairly certain that, while the Imperials might have been unable to resist the urge to put that one into immediate operation, they also would have been sure to find a less important one somewhere that they could take apart to figure out how the damn things worked…]
It Is Become Death, the Destroyer of Worlds
“I don’t believe it’s that the Transcend doesn’t want the competition. Of course, that’s exactly what I would say in either case, so there’s very little point in my denial; I’ll leave that up to your viewers to decide.”
“It’s not a matter of freedom of information. There’s no secret contract or midnight visits from anyone who doesn’t exist keeping us from publishing. But I don’t think it needs any conspiracy, or even any collaboration, to explain the Consensus.”
“Given the number of messy, spectacular, and civilization-terminating failures that we’ve seen, historically, even among people who’ve worked out the science for themselves – how enthusiastic would you be about handing out to amateurs the secrets of computational theogeny?”
– Academician Alwyn Steamweaver, ICIN interview on the Corícal Consensus