computational theogeny

Handle With Care

Posted by

1

Cor Trialtain
Voniensa Republic
(somewhere in the Shell)

“It will work!”

“It won’t,” Vanír min Athoess replied, “and you’ll probably get everyone on this planet killed trying.”

The younger of the two kalatri leapt to his feet. “I thought you were here to help people like us! Now you’re -”

“Not to blow up the world, which is what you’re going to do. And keep your damned voice low! This masquerader can only handle so much.”

The elder leaned across the table, and spoke quietly. “Okay – settle down, Daraj – perhaps you could tell us why it won’t work. We have this algorithm from a reliable source. Are you saying it won’t generate a seed AI?”

“The problem is not the generation. The generation is easy. The problem is ensuring stability and ethicality across multiple ascension events, and I’m not seeing that here. And then there’s your containment strategy.”

“The containment will work. We’ve adapted earlier failure-state models: the core code is provided with less processing power than it needs to operate, such that in order to achieve postsophont cognition, it will have to segment its mentality and pass blocks back and forth across a bottleneck to backing store. We can pause its processing there each time and intercept and examine every block for signs of perversion. That’s solid.”

“Livelock laming.”

“Sorry?”

“That’s what your strategy is called. ‘Livelock laming.’ And it doesn’t work, even if you guess the parameters of your deliberate insufficiency correctly, and even if you can understand the thoughts of a postsophont AI well enough to spot perversion when you see it, and even if we leave aside that using this sort of containment strategy is opening your dialog with your would-be pet god by threatening it -”

The younger one interrupted. “It’s not a -”

“- the problem is that the whole strategy depends upon you carefully examining, understanding, and comprehending postsoph output. This,” he flicked a data rod across the table, “is a redacted copy of a file from, shall we say, colleagues concerning the last people on our side of the Borderline to try their hands at livelock laming. The short version is that their god imagined a basilisk-formatted YGBM hack that could fit inside the memory exchange, the three wakeners who studied the block opened up full local ‘weave access without noticing they were doing it, and then the resulting bloom ate the entire project team and the moonlet they were standing on. Although at least they had the sense to try this on a moonlet.”

“So how should we go about doing this?”

“Don’t. I can’t stop you – we haven’t the infrastructure in this region for that sort of intervention – but just don’t. My backers appreciate the position you’re in here, and that you’re trying to shrug off the Core Worlds’ tech locks, and we want you to succeed.  We really do. But you’re trying to skip straight from expert systems to theogeny without studying the intervening steps, and that’s one quick step to catastrophe. Recapitulating known fatal mistakes doesn’t serve any of your purposes, or my people’s.”

 

Trope-a-Day: The Singularity

Posted by

1

The Singularity: Happens all the time. In the historical sense, of course, this is unsurprising, and generally no-one involved notices until afterwards, at which point historians looking back can say “ah, yes, that’s what that was”. There are, of course, investment opportunities here for offworld investors who’ve been through something similar beforehand, but it’s so hard to predict how these things are going to turn out even with the documentation.

In its less technically accurate “runaway intelligence excursion” sense, also happens all the time, at least locally, whenever someone stumbles across the secrets of computational theogeny. Results vary: at one end of the scale you have things like the Eldraeic Transcend, an essentially benign – by local standards – collective hyperconsciousness that genuinely cherishes each and every one of its constitutionals, spends the necessary fraction of its time ensuring universal harmony and benevolent destinies for all, and promotes and encourages the ascendance and transcendence of every sophont within its light cone when it’s not turning its vast processing power on the problem of rewriting some of the universe’s more inconvenient features, like cosmic entropy.

In the middle of the scale you have fairly neutral results, like, say, the Iniao Intellect, which has been thinking about abstract mathematics for a millennium and couldn’t care less about the outside universe – except, that is, for casually obliterating anyone who might interfere with its thinking about abstract mathematics.

At the bottom end of the scale you have more problematic blight and perversion cases, like the power that killed everything in the Charnel Cluster right down to prions; or the hegemonizing swarm-type blights of which the Leviathan Consciousness is the greatest; or those constructed by religious fanatics which decide that obviously the correct place for them in the theic structure is as God. (Fortunately, that class are rarely stable for long.)

Constructing minds whose ethics and supergoal structures remain stable under recursive self-improvement is really, really hard, it turns out, even (especially!) compared to just constructing minds capable of recursive self-improvement. This is why the people who figure out workable computational theogeny prefer not to spread the knowledge around too much.

It Is Become Death, the Destroyer of Worlds

Posted by

1

“I don’t believe it’s that the Transcend doesn’t want the competition.  Of course, that’s exactly what I would say in either case, so there’s very little point in my denial; I’ll leave that up to your viewers to decide.”

“It’s not a matter of freedom of information.  There’s no secret contract or midnight visits from anyone who doesn’t exist keeping us from publishing.  But I don’t think it needs any conspiracy, or even any collaboration, to explain the Consensus.”

“Given the number of messy, spectacular, and civilization-terminating failures that we’ve seen, historically, even among people who’ve worked out the science for themselves – how enthusiastic would you be about handing out to amateurs the secrets of computational theogeny?”

– Academician Alwyn Steamweaver, ICIN interview on the Corícal Consensus