One comment refuses to leave me: https://www.lesswrong.com/posts/DfrSZaf3JC8vJdbZL/how-to-make-superbabies?commentId=C7MvCZHbFmeLdxyAk

The commenter makes and extended tortured analogy to machine learning… in order to say that maybe genes with correlations to IQ won’t add to IQ linearly. It’s an encapsulation of many lesswrong issues: veneration of machine learning, overgeneralizing of comp sci into unrelated fields, a need to use paragraphs to say what a single sentence could, and a failure to actually state firm direct objections to blatantly stupid ideas.

My favorite comment in the lesswrong discussion: https://www.lesswrong.com/posts/DfrSZaf3JC8vJdbZL/how-to-make-superbabies?commentId=oyDCbGtkvXtqMnNbK

It’s not that eugenics is a magnet for white supremacists, or that rich people might give their children an even more artificially inflated sense of self-worth. No, the risk is that the superbabies might be Khan and kick start the eugenics wars. Of course, this isn’t a reason not to make superbabies, it just means the idea needs some more workshopping via Red Teaming (hacker lingo is applicable to everything).

Soyweiser has likely accurately identified that you’re JAQing in bad faith, but on the slim off chance you actually want to educate yourself, the rationalwiki page on Biological Determinism and Eugenics is a decent place to start to see the standard flaws and fallacies used to argue for pro-eugenic positions. Rationalwiki has a scathing and sarcastic tone, but that tone is well deserved in this case.

To provide a brief summary, in general, the pro-eugenicists misunderstand correlation and causation, misunderstand the direction of causation, overestimate what little correlation there actually is, fail to understand environmental factors (especially systemic inequalities that might require leftist solutions to actually have any chance at fixing), and refuse to acknowledge the context of genetics research (i.e. all the Neo-Nazis and alt righters that will jump on anything they can get).

The lesswrongers and SSCs sometimes whine they don’t get fair consideration, but considering they take Charles Murray the slightest bit seriously they can keep whining.

That was literally the inflection point on my path to sneerclub. I had started to break from less wrong before, but I hadn’t reached the tipping point of saying it was all bs. And for ssc and Scott in particular I had managed to overlook the real message buried in thousands of words of equivocating and bad analogies and bad research in his earlier posts. But “you are still crying wolf” made me finally question what Scott’s real intent was.

I normally think gatekeeping fandoms and calling people fake fans is bad, but it is necessary and deserved in this case to assume Elon Musk is only a surface level fan grabbing names and icons without understanding them.

This is a good summary of half of the motive to ignore the real AI safety stuff in favor of sci-fi fantasy doom scenarios. (The other half is that the sci-fi fantasy scenarios are a good source of hype.) I hadn’t thought about the extent to which Altman’s plan is “hey morons, hook my shit up to fucking everything and try to stumble across a use case that’s good for something” (as opposed to the “we’re building a genie, and when we’re done we’re going to ask it for three wishes” he hypes up), that makes more sense as a long term plan…

It’s not all the exact same! Friendship is Optimal adds in pony sex

There’s also a whole subreddit from hell about this subgenre of fiction: https://www.reddit.com/r/rational/

/r/rational isn’t just for AI fiction, it also claims includes anything with decent verisimilitude, so stuff like The Hatchet and The Martian show up in its recommendation lists also! letting it claim credit for better fiction than the AI stuff

Oh no, its much more than a single piece of fiction, it’s like an entire mini genre. If you’re curious…

A short story… where the humans are the AI! https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message Its meant to suggest what could be done with arbitrary computational power and time. Which is Eliezer’s only way of evaluating AI, by comparing it to the fictional version with infinite compute inside of his head. Expanded into a longer story here: https://alicorn.elcenia.com/stories/starwink.shtml

Another parable by Eliezer (the genie is blatantly an AI): https://www.lesswrong.com/posts/ctpkTaqTKbmm6uRgC/failed-utopia-4-2 Fitting that his analogy for AI is a literal genie. This story also has some weird gender stuff, because why not!

One of the longer ones: https://www.fimfiction.net/story/62074/friendship-is-optimal A MLP MMORPG AI is engineered to be able to bootstrap to singularity. It manipulates everyone into uploading into it’s take on My Little Pony! The author intended it as a singularity gone subtly wrong, but because they posted it to both a MLP fan-fiction site in addition to linking it to lesswrong, it got an audience that unironically liked the manipulative uploading scenario and prefers it to real life.

Gwern has taken a stab at it: https://gwern.net/fiction/clippy We made fun of Eliezer warning about watching the training loss function, in this story the AI literally hacks it way out in the middle of training!

And another short story: https://www.lesswrong.com/posts/AyNHoTWWAJ5eb99ji/another-outer-alignment-failure-story

So yeah, it an entire genre at this point!

Short fiction of AGI takeover is a lesswrong tradition! And some longer fics too! Are you actually looking for specific examples and/or links? Lots of them are fun, in a sci-fi short form kind of way. The goofier ones and cringer ones are definitely sneerable.

I mean, if you play on the doom to hype yourself, dealing with employees that take that seriously feel like a deserved outcome.

Some nitpicks. some of which are serious are some of which are sneers…

consternating about the policy implications of Sam Altman’s speculative fan fiction

Hey, the fanfiction is actually Eliezer’s (who in turn copied it from older scifi), Sam Altman just popularized it as a way of milking the doom for hype!

So, for starters, in order to fit something as powerful as ChatGPT onto ordinary hardware you could buy in a store, you would need to see at least three more orders of magnitude in the density of RAM chips—​leaving completely aside for now the necessary vector compute.

Well actually, you can get something close to as powerful on a personal computer… because the massive size of ChatGPT and the like don’t actually improve their performance that much (the most useful thing I think is the longer context window?).

I actually liked one of the lawfare AI articles recently (even though it did lean into a light fantasy scenario)… https://www.lawfaremedia.org/article/tort-law-should-be-the-centerpiece-of-ai-governance . Their main idea is that corporations should be liable for near-misses. Like if it can be shown that the corporation nearly caused a much bigger disaster, they get fined in accordance with the bigger disaster. Of course, US courts routinely fail to properly penalize (either in terms of incentives of in terms of compensation) corporations for harms they actually cause, so this seems like a distant fantasy to me.

AI has no initiative. It doesn’t want anything

That’s next on the roadmap though, right? AI agents?

Well… if the way corporations have tried to use ChatGPT has taught me anything, its that they’ll misapply AI in any and every way that looks like it might save or make a buck. So they’ll slap an API to a AI it into a script to turn it into an “agent” despite that being entirely outside the use case of spewing words. It won’t actually be agentic, but I bet it could cause a disaster all the same!

I saw people making fun of this on (the normally absurdly overly credulous) /r/singularity of all places. I guess even hopeful techno-rapture believers have limits to their suspension of disbelief.

First of all. You could make facts a token value in an LLM if you had some pre-calculated truth value for your data set.

An extra bit of labeling on your training data set really doesn’t help you that much. LLMs already make up plausible looking citations and website links (and other data types) that are actually complete garbage even though their training data has valid citations and website links (and other data types). Labeling things as “fact” and forcing the LLM to output stuff with that “fact” label will get you output that looks (in terms of statistical structure) like valid labeled “facts” but have absolutely no guarantee of being true.

His replies have gone up in upvotes substantially since yesterday, so it looks like a bit of light brigading is going on.

Reddit can be really hit or miss, but I’m glad subredditdrama and /r/wikipedia aren’t buying TWG’s bullshit. Well, some of the /r/wikipedia assume TWG is merely butthurt over losing edit wars as opposed to a more advanced agenda, but that is fair of them.

I chose to have children, be a father and a husband, live an honest industrious life as an example to my offspring, and attempt to preserve my way of life through them.

Wow, just a few words off the 14 words.

I find it kind of irritating how someone that doesn’t familiarize themselves with white supremacists rhetoric and methods might manage to view that phrase innocuously. But it really isn’t that hard to see through the bullshit once you’ve familiarized themselves with the most basic dog whistles and slogans.

Wow… I took a look at that link before reading the comments/explanations here, and I was briefly confused why they were hating on him so much, before I realized he isn’t radical right wing enough for them.

Eh, you’re a gay furry ex-Mormon (which is like a triple strike against you in my book) but I still like you well enough.

It is almost sad seeing TWG trying to appeal to these people that fundamentally hate him… except he could just admit themotte is a cesspit and abandon it. But that would involve admitting that sneerclub (and David Gerard specifically) was right about the sort of people that lurked around SCC and later concentrated within themotte, so I think he’s going to keep making himself suffer.

TW knows about the propaganda war, but has very different objectives to you. Much harder to balance ones too: he needs enough Progress for surrogate gaybies, but not too much that white gay guys can’t get the good lawyer jobs.

Wow, I feel really gross agreeing with a motte poster, but they’ve called out TWG pretty effectively. TWG at least knows he needs things progressive enough he doesn’t end up against the wall for being gay, ex-Mormon and furry (as he describes himself), yet he wants to flirt with the alt-right!

and in case I was in danger of forgetting what the motte really is…

Yes, we’ve all thrown our hat in the ring in different ways. I chose to have children, be a father and a husband, live an honest industrious life as an example to my offspring, and attempt to preserve my way of life through them.

sure buddy, you just need to “secure the future for your people and your children”… Yeah I know the rest of the words that go in that slogan.

I’m almost certain I’ve seen EY catch shit on twitter (from actual ml researchers no less) for insinuating something very similar.

A sneer classic: https://www.reddit.com/r/SneerClub/comments/131rfg0/ey_gets_sneered_on_by_one_of_the_writers_of_the/

I am probably giving most of them too much credit, but I think some of them took the Bitter Lesson and learned the wrong things from it. LLMs performed better than originally expected just off context, and (apparently) scaled better with bigger model and more training than expected, so now they think they just need to crank up the size and tweak things slightly (i.e. “prompt engineering” and RLHF) and don’t appreciate the limits built into the entire approach.

The annoying thing about another winter is that it would probably result in funding being cut for other research. And laymen don’t appreciate all the academic funding that goes into research for decades before an approach becomes interesting and viable enough to scale up and commercialize (and then overhyped and oversold before some more modest practical usages become common, and relabeled as something other than AI).

Edit: or more cynically, the leaders and hype-men know that algorithmic advances aren’t an automatic dump money in, get out disruptive product process, so they don’t bother putting as much monetary investment or hype into algorithmic advances. Like compare the attention paid towards Yann LeCunn talking about algorithmic developments vs. Sam Altman promising grad student level LLMs (as measured by a spurious benchmark) in two years.