One of the classic sequence posts plays with the concept: https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message

But in a uno reverse twist, its the humans that are a metaphor for AI, with them manipulating out thinking the aliens from within the simulation.

Someone on Scott Aaronson’s blog suggested math will become an empirical field now, where they run “experiments” by asking AI for lean proofs.

Scott, is this a turning point where mathematics gets off its well trodden path of logic and becomes more like an empirical science? Heck, people were arguing if math is invented or discovered, this settles it. Math is discovered – by AI! From now on forth, Mathematicians will become scientists, wielding AI as a telescope, pointed towards new and disturbing alien worlds in the Platonic realm.

So yeah, it seems like this idiotic view is pretty popular among boosters.

The theory on the thread that

My guess is that this represents a shift in focus at OpenAI away from math: they had their big meaningless trophy prize with the Navier-Stokes solution, they appear to have distanced the competition (Anthropic) and the public bought the notion that “math is solved”

seems like a plausible explanation? I’ll guess we’ll if OpenAI moves on now that they’ve “solved” math or if they follow up with a few more drops of “proofs”.

True, but its a different form of disrespect, where they outwardly steal the style and aesthetics of the discipline to try to claim its legitimacy. Like the endless “whitepapers” pumped out by cryptobros.

I do not see the point of math for AI alone, since I do not think AI is or will ever be conscious.

Rationalists (and in general those with fear/hope or a technological singularity) have the idea that once the AI is smart enough, it will be able to recursively self-improve in an exponential process. Vibe-coding hasn’t made this come true, so they are trying to make it happen by shoving enough of the right sort of knowledge into the AI. Obviously being smart and being mathey go together, so if the AI just gets good enough at math, surely bootstrapping to godhood is just around the corner!

To them, the AI is not here for us, but we are here to serve the AI.

The techbros are giving mathematicians the humanities treatment. (I.e. they don’t respect the entire profession despite its immense importance to actually understanding reality.) Before, mathematics had sort of gotten half a pass for being rigorous and adjacent to stuff techbros do respect, like computer science. But, given just half a whiff of the idea that they may no longer need human mathematicians, they are ready to accept utter slop…

Place like /r/singularity and hackernews are eating up the hype and buying into it. Even with mathematicians explaining carefully the errors in the proofs and why they are badly written and useless to actually progress in mathematics the boosters are treating that like sour grapes or irrational fear. If mathematicians tried outright ignoring it, OpenAI’s narrative would go even more unchallenged.

You know how AI proofs are supposed to be credible because of formalization with Lean?

There is another problem… apparently Lean4 hasn’t actually been fully formally verified. And in fact could have more bugs lurking in it, maybe even some foundational problems… Lesswrong post explaining the problems.

OpenAI can more easily market that their AI is intended to fully replace mathematicians, and that this is only the herald of their general superintelligence (and not that math is the only thing working out for them).

Yeah, this has been driving me crazy… reading places like hackernews or r/singularity, I keep seeing people assuming mathematicians and just bitter or engaging in sour grapes and that is why they are complaining about the “proofs” OpenAI dumped out. Like it is just some small niggling detail that the proofs can’t be read by humans and that Lean may have bugs and that some of OpenAI’s proofs have already been shown to have critical mistakes.

Or cite it, but in the most passive aggressive way you can get away with.

…You should check the literature really carefully to make sure that whatever proof the AI “generated” doesn’t actually already exist somewhere and it regurgitated it without proper accreditation.

Yeah lesswrong is weird about China. Its an ongoing thing. It alternates between fear (that China won’t appreciate the risk of making AI Satan or will beat the US to making AI God) and occasional hope (that they might manage to “safety-pill” China and China will find the magic formula to AI alignment). I pulled together some posts previously here: https://awful.systems/post/4103825

If it had been only a year they’ve been saying that, I might almost buy the logic. But it has been 4 years since Chat-GPT came out and they’ve been making the same promises the entire time.

Something of a tangent… thinking back to 2022-2023 my expectations for what the LLM companies would actually achieve by now were actually too high? Like I would have thought they would have figured out a better architecture for gluing functionality and symbolic logic to LLMs than just calling tools with the output text stream and feeding the tool outputs into the context window.

They actually both converge to a retelling of SCP-8008.

Not only that, they view commonsense moral and ethical reasoning as a PR constraint, i.e. look at the discourse around the hiring of Caroline Ellison to an EA org that handles significant sums of money. Lot of commenters were merely framing it as a PR problem and not a basic commonsense ethics problem.

Considering how much compensation he gets a MIRI, he could afford to hire an artist out of his own pocket and he would end up with something better and not be funding the technology that he claims is going to kill us all…

The thing about the “yet you participate in society, curious!” meme is that you don’t really get the option to not participate in society. you do however get the option to not participate in AI, and there are plenty of people happily taking that option

Yeah, rationalist utterly fail at understanding collective action, or really anything besides lib-brained centrist methods of action. Well, maybe even lib-brained centrist is giving them too much credit, because even milquetoast liberals will occasionally avoid using products for ethical reasons. See Only Law Can Prevent Extinction, where Eliezer basically explains how all other action besides an international total ban on AI and GPUs is pointless. (I think he is trying to avoid people bombing data centers, but Elizer’s “logic” is also implicitly against stuff like going to town halls to complain about data centers.)

This has been talked about in the reddit sneerclub, but not yet on awful systems (as far as I could tell with a quick search on this and last week’s stubstack)…

Caroline Ellison (the convicted FTX fraudster) got hired for an EA position dealing with financial allocations!

Reddit sneerclub discussion

EA Forums Discussion surprisingly (by the low low bar we have for EA) this announcement was heavily downvoted and all the top comments clearly and frankly denounce Manifund for hiring Ellison, accurately calling out the sheer violation of common sense and the obviously unethical nature of hiring someone because they previously directed money your way that turned out to be stolen. (The Manifund person outright admits this as a motivation!) Not so surprisingly, less upvoted, but still upvoted comments do some hand-wringing about the optics (as opposed to the clear ethical and common sense issues) and some even outright call for forgiveness for Caroline!

The lesswrong discussion is less surprising (as in it is full classic lesswrong fallacies). It starts with considering the optics, and some posters, such as habryka, outright question the existence of any such ethical rule that hiring Caroline might violate! (If there isn’t a rule anywhere, its only because “don’t hire convicted fraudsters for positions dealing with a lot of money” is such blatant common sense.)

Bonus: A galaxy brain take: maybe since Caroline has already messed up so massively, that means she actually has the best experience!

A lesswronger asks are we rationalfic protagonists the baddies? https://www.lesswrong.com/posts/FuGfR3jL3sw6r8kB4/richard-ngo-s-shortform?commentId=uDuzmfMEvEqpyApLh

tldr; rationalfic has a very common trend of the protagonist gaining and using overwhelming power to radically reform the world. This is almost (with a few notable exceptions) portrayed as clearly unambiguously good thing.

My take: Don’t get me wrong, the Wizarding World (for example), as canonically portrayed needs some very strong reforms if not an entire revolution. But rationalfic almost never portrays the slow hard work of building support networks and alliances and developing a materialist theoretical understanding of how to reform society, as opposed to a lone (or small friend group) rationalist hero finding some overwhelming magical or technological advantage they can use to single-handedly take control and use their rationalist intellect to unilaterally fix everything. Part of it is the normal disconnect of fiction to the real world were it is more narrative satisfying (and easier to write) to have a central protagonist the solves the major problems or is at least directly involved with them, and rationalfic involves that protagonist gaining even more agency than they canonically do. The problem is that rationalist take this attitude back into real life, and so end up idolizing mythologized techbro billionaires or venture capitalist or the myth of the lone genius scientist/inventor.

Also, quality sneer in the replies, “rational” teletubbies: https://tomasbjartur.bearblog.dev/rational-teletubbies/