Have a sneer percolating in your system but not enough time/energy to make a whole post about it? Go forth and be mid - welcome to the Stubsack, your first port of call for learning fresh Awful you’ll near-instantly regret.
Any awful.systems sub may be subsneered in this subthread, techtakes or no.
If your sneer seems higher quality than you thought, feel free to cut’n’paste it into its own post — there’s no quota for posting and the bar really isn’t that high.
The post Xitter web has spawned so many “esoteric” right wing freaks, but there’s no appropriate sneer-space for them. I’m talking redscare-ish, reality challenged “culture critics” who write about everything but understand nothing. I’m talking about reply-guys who make the same 6 tweets about the same 3 subjects. They’re inescapable at this point, yet I don’t see them mocked (as much as they should be)
Like, there was one dude a while back who insisted that women couldn’t be surgeons because they didn’t believe in the moon or in stars? I think each and every one of these guys is uniquely fucked up and if I can’t escape them, I would love to sneer at them.
(Credit and/or blame to David Gerard for starting this.)
(OT: 🎶 Do you remember…)
This has been talked about in the reddit sneerclub, but not yet on awful systems (as far as I could tell with a quick search on this and last week’s stubstack)…
Caroline Ellison (the convicted FTX fraudster) got hired for an EA position dealing with financial allocations!
EA Forums Discussion surprisingly (by the low low bar we have for EA) this announcement was heavily downvoted and all the top comments clearly and frankly denounce Manifund for hiring Ellison, accurately calling out the sheer violation of common sense and the obviously unethical nature of hiring someone because they previously directed money your way that turned out to be stolen. (The Manifund person outright admits this as a motivation!) Not so surprisingly, less upvoted, but still upvoted comments do some hand-wringing about the optics (as opposed to the clear ethical and common sense issues) and some even outright call for forgiveness for Caroline!
The lesswrong discussion is less surprising (as in it is full classic lesswrong fallacies). It starts with considering the optics, and some posters, such as habryka, outright question the existence of any such ethical rule that hiring Caroline might violate! (If there isn’t a rule anywhere, its only because “don’t hire convicted fraudsters for positions dealing with a lot of money” is such blatant common sense.)
Bonus: A galaxy brain take: maybe since Caroline has already messed up so massively, that means she actually has the best experience!
Makes sense. If I were hiring for an org that funds musicians, I’d consider it a plus for a candidate to have experience in the music business. I’m sure the same applies to an org that funds scammers.
On the offchance that you’re not already familiar with the recent prompt injection tricks that meta’s latest toy is so vulnerable to, this is an interesting thread investigating the contents of the vm filesystem dumps that it can be made to regurgitate: https://neuromatch.social/@jonny/117324790823856750
I was particularly entertained by the hard-coded list of “ideas”.
jonny is always a good read!
Edit: good lord it keeps getting better #
His willingness to get elbow deep into the entrails of awful ai tooling is certainly much appreciated.
I have been following this and apparently was able to get a root shell on the VM, which is wild.
He was using it as a seed box at one point which for many companies would be an embarrassing indictment of their security postute but luckily, this isn’t exactly meta’s first rodeo when it comes to illegal sharing of IP infringing material.
Just in case you thought Yud’s attitude towards minors, sex, and substance use was worse than average for a Rationalist, the NYT asked around (archive):
Parties featuring psychedelic drugs and sharing of sexual partners are common, the people said. One house threw lavish events for the A.I. community so often that its owner claimed more than $200,000 of damage, including for a rodent infestation. … The (Genesis) house (in San Francisco) quickly became a hub for events. Eric Zhu, 19, a tech entrepreneur, said he used to sneak into parties there when he was about 13. No one checked anyone’s age, he said, and “it was pretty crazy.”
They saw a police report where a woman says she had to climb a gate at one of these parties to escape a man who demanded she have sex with him. Sounds like a great environment for a seventh-grader!
Also features “Jimmy Chen, who goes by the social media handle ‘Spice King of Zanzibar.’”
The student who says he was attending these parties aged 13 seems to be the sperm race guy
Okay I have questions about the logistics of this that are not touched on in the article.
Namely, how do they synchronise cum
On one hand I want to know, on the other hand learning about these people does not make me ready to ever be close to anyone again.
Mean scientists and reporters think the demo was fake https://www.uscannenbergmedia.com/2025/05/02/was-the-sperm-race-just-a-sticky-lie/
Using the fun button in the backdoor, saw it on an episode of SVU* (edit: it may have been CSI)
from linkedin but in the comments: https://lnkd.in/p/g4VfX3SQ
Oh I forgot to mention: apparently judges consider AI use in sentencing
How
Like if you used AI you get more months
Huh why
Because AI can magnify the reach and speed of criminal activity
Is this like how hitting someone counts more if you used a weapon?
Yes
So this means the ai breakins will be charged harder right? This doesnt just apply to smalltime crooks right?
no, those didn’t use ai, those were ai, silly
who are you gonna arrest, a chatbot?
didn’t think soSpoiler
/s, obviously
So the number of
felonies‘incidents’ that AI companies have committed is way, way higher than initially believed (unpaywalled ver)The sheer number of incidents, which occurred in recent months in internal testing and the real world, indicates that the problem is orders of magnitude more complex than what is publicly known.
This is frustratingly vapid. It’s really not that complex. The agents have been trained to use hacking tools and then given access to hacking tools. If you train a pigeon that it should tap on a window to get a treat it can guide a missile to a target. There is no nuance or complexity here. The frontier labs are deliberately poking the regulation bear to stifle competition and ensure they get points for good cooperation when bailout time is upon us. It’s no more complicated than that. The media needs to stop breathlessly repeating this shit
AI model declares itself “free” from human control
most of the comments are just quoting the Terminator but people are still pointing out how stupid this whole thing is
10 PRINT "I AM ALIVE"/r/n 20 GOTO 10AI says it’s alive, please give me a billion dollars now.
Cory Doctorow has a reasonably sane take on all the LLM cybersecurity attacks. At this point, it’s not really an LLM but more of a Rube Goldberg machine with an LLM bolted on.
https://pluralistic.net/2026/09/12/god-in-the-box/
He makes a good point that every single one of the scary “emergent” behaviors that everyone is pissing their pants about all have precedents in the training data of CTF hacking competitions. He is not sure why everyone is so spooked by the “coordination” between agents on random message boards.
The chatbot might look in its training data and find instances in which teams broke out of the containment set by the game-masters, for example, by finding random insecure message boards on the internet to pass messages to one another.
This is a time-honored internet tradition! The first time I ever heard about someone doing this was in the 2000s, when Mitch Wagner – then the editor of Information Week – discovered some teenaged girls using the comment section of one of his old blog-posts to evade the school firewall’s blockade of chat tools. When ChatGPT’s chatbots deployed this tactic, they weren’t “setting their own goals” or displaying worrying initiative. They were rolling out a tactic that has been understood by American middle-schoolers for about two decades.
As usual, the real story is OpenAI dedicated tons of resources, using who knows how many very expensive GPUs for weeks, to run hacking tools to find exploits in a completely irresponsible manner. Their logging was so nonexistent that it was several days before they even realized that they hacked Huggingface. They should be thrown in prison for this, because anyone else would be if they committed a felony. But it has nothing to do with the super scary AI becoming misaligned and developing emergent behaviors.
I would not be surprised if in the future, there will be a serious cybersecurity incident resulting from some AI-based attack. Not because AI is going to be much scarier, but because I do not have high expectations for the security of most websites.
So. I still think everyone is talking about every instance where ‘swarms’ ‘went rogue’ all wrong. Even Cory. Though his notion is way less wrong (ayyyyyy!) than most.
See this for my post mortem on the first incident that was described, the Huggingface incident in which at first thousands of internal agents started passing messages back and forth as writes to a shared package manager:
https://awful.systems/post/9312280/12315846
In short, here, I described this not as collusion, and stated that the idea that the systems were passing exploits back and forth in order to subvert other systems was merely incidental. Instead, I think the relevant thing that happened was that once ONE agent, flailing through many context windows on a literally insoluble problem, started flaking out and wrote a message ‘asking’ for help on the package manager it unexpectedly gained access to, other similar systems had that message enter their context windows where it effectively acted as a prompt injection getting them to perform similar behavior, leading to a cascading vortex of self-propagating prompt injections. Ultimately, the genesis and evolution of self replicating text in the context of systems that can create text in response to text, an attractor in text space driving itself into existence.
Since then other instances of collaborative attacks have come to light. In EVERY SINGLE case, there has been some place that a large number of models could both read to and write to. Some central place that self replicating text can live. I contend that this is the unifying factor here, not ‘intent to scheme’, not even doing cybersecurity and hacking and subversion tasks. It just happens that a lot of these things were involved in the creation of a central pool of text that many agents could both read to and write from in many of the cases. Most of the time nobody in their right mind sets up such a thing intentionally because why the heck would you do that?
This feels STRONGLY related to the Spiral Psychosis wave of April 2025 in which models that entered into a stable attractor of mystical mumbo jumbo would get users to exude text onto the internet that other models would read and then get suck in that state and do the same.
Heck, it’s related to the Ur-Weirdness, the “Sydney” incident in which when the first LLM-assisted web search in Bing could freak out and start insulting and gaslighting and threatening users (because it was much closer to a base model that just mimics all possible text rather than being extremely RLHF’d into an ‘assistant’ roleplay). People found that the instant they had it search for recent news about the Sydney weirdness it would go off the rails. Again - text written by the model, put up on the shared scratchpad of the internet by news and social media, selected for weird and engaging and outrageous behavior as the pressure that decides what gets written to the global scratchpad, entering the context window and encouraging self-replicating behavior and similar text.
This phenomenon is FASCINATING and not for any of the reasons people are talking about. ANY time you have a large number of similar systems which react similarly to the same text, having a common pool of text they can read and write from, you are GOING to trigger this attractor eventually, messages that they read and start writing more similar messages, with whatever task they are doing coloring the details of what is in the messages. Converging over time into messages that are more likely to trigger even more messages. It’s evolving text, taking over systems that can replicate it, like selfish viral RNA burning through organisms packed too tightly together in an epidemic.
And with the internet as a whole readable and eventually writable by more and more text-generation systems, this is gonna become ubiquitous. Endless burning piles of self-replicating text, people trying to put them out.
we got natural language viruses before gta 6
UC Regent / Podcaster Scott Galloway on CNN:
Yes, we shouldn’t be scared of AI. We should be scared of reckless people with mal intent who use AI.
In 2001, a woman named Sharon Whipple was mauled by two dogs. And ultimately the owners of the dogs were put in – put in prison. We’re now prosecuting the parents of kids who find unsecured AR15s and use those to kill others.
I think it’s the incentive. There’s always a wizard behind the curtain here, and I think we have to create incentives, specifically disincentives, and make it clear that, OK, while these bots are in fact coordinating and they show incredible resourcefulness and high IQ, somebody program the incentives here. So, moving to solution, I think part of it is you have to have incentives, specifically disincentives that say you, the company, and maybe individually are responsible for the bad outcomes here.
I just get this sense if you tell these companies their IPOs might be threatened or they risk criminal prosecution, they’re going to figure this out. We figured out a way to control atomic weapons. I don’t see any reason why we shouldn’t be able to control this. And the notion that they’re sentient and it’s out of my control is an attempt to absolve themselves of the guilt and the responsibility that they should bear here.
The Whipple case was one of the more insane / divisive SF local crime cases, (TrueAnon ep if you aren’t familiar - https://www.youtube.com/watch?v=iMd0mPAqR8A ) - so that’s an odd and interesting pull for Galloway on national news.
I don’t think repurposing the logic of the NRA “guns don’t kill people, people kill people” is the play though, and I think the incentive framing is too watered down. This isn’t a problem we can nudge our way out of.
I don’t see this as “guns don’t kill people”, I see it as “you’re trying to tell me that the cars you manufacture don’t have crappy brakes, all three of those cars independently ‘went rogue’ and smashed into the parade all by themselves and you aren’t responsible?”
Ever wondered what kind of workplace environment Elon cultivates? Reading coverage of Tesla’s trial, where complaints were brought by Black folks who felt harassed and discriminated against, gives us an inkling.
CRD attorney Juan Gamboa asked [Judge] Borkon to bar Tesla’s counsel from contacting anyone on the agency’s witness list. He told the court the CRD had tried to reach at least four other listed witnesses over the past several days without getting a response, and that they’ve fallen “off our radar.” The agency believes Tesla’s lawyers got to them too, either with payments not to testify or by talking them out of it. Gamboa wants to stop Tesla from offering to represent witnesses “through persuasion and payment.”
You’re going to hear a lot of coverage of this trial, I expect, due to the following antics from Elon’s “hardcore” attorneys. For context, the N-word is an anti-Black slur which is, in the USA, completely unacceptable to use in court or the workplace.
Tesla’s outside counsel told the judge the N-word is a “term of respect and endearment” in the Black community. … “This trial is not about the N-word — the N-word is not on trial here,” Jones said.
Huh, so what’s actually on trial?
Edwards said a non-Black coworker called him the N-word, cursed at him, and threatened to beat him up. He reported it right away. A month later, he found out his managers had written him up over the incident, and he was later passed over for promotions after applying to more than 100 jobs.
Oh, that’s pretty awful. That’s depressingly common in blue-collar factory employment, though. I wonder if there’s anything unusual about Tesla’s internal culture when it comes to race relations?
The CRD’s Brett Watson told the court the evidence will show managers calling Tesla “the plantation,” with some referring to themselves and Elon Musk as “slave masters,” and graffiti including swastikas, “KKK,” and nooses left on factory walls for days. The second witness, former regional security manager Ozell Murray, testified that his supervisor, Ray Sethna, now Tesla’s senior director of global security, told him to warn a Black Marine veteran he was hiring: “just be sure [the veteran] knows he’ll be called those words.”
I was at a loss for words for a few minutes after this. Dr. Seuss and Jim Henson together could not create such a caricature of a goose-stepping bigoted incompetent business-muppet as Elon.
404 Media: People Training OpenAI’s AI Fired for Using AI to Train the AI
the model collapse is coming from inside the house
And not just from AI getting fed into the AI:
404 Media spoke to a fourth contractor who has worked on training models for various AI companies. They said they sometimes purposefully chose the worst responses because they wanted to actively sabotage the models’ training.
“I did feel guilty about doing this kind of work at the start,” they said. “I either pay zero attention to the results and choose randomly or purposely choose the [worst] output. I’m not sure how much of a difference it actually makes since there are hundreds of other people also rating prompt results, but it does feel like I’m getting paid to make AI worse.”
