I usually don’t even understand the lingo they use. “Open-weighted” is the most recent one, then it usually goes down to specific “models” that everybody is supposed to know about.
These are my thoughts (I will stick to the vague “it” for now, but of course therein lies another question: “and how does all this apply to various specialised AIs”):
- Is it really feasible to run it 100% locally? I know there’s plenty of people with very powerful rigs indeed, but still. Or are 99% of these people really saying “it would, in theory, be possible to run that locally, therefore your concerns are invalid”?
- If yes to the previous: the software doesn’t come from nowhere and ultimately still relies on gas-turbine-powered datacenters and stolen IP and stolen personal data, no?
If what I wrote above is true, what exactly are people arguing when they say it’s still possible to use LLMs ethically or true to FOSS philosophy, because … ???
edit
Thanks to all who answered.
I guess it’s my fault for asking several questions in one, but this thread has attracted exactly the type of people I’m writing about; several even used the term “open-weighted models” without explaining it.
Asking to get arguments explained, I got more arguments instead.
You need some decent hardware to do it, but yes one can absolutely run capable local models. The 7-8B models run fast and are probably decent for light coding, but they make piss poor chat bots. 27-36B models get much better outputs but take more hardware, and still not great in a chat interface (but far better than 8B). They run slow enough on my MacBook that it’s just a toy for fun, despite unified memory. I aspire to having a local AI server then can run models in the 100B-350B range. It’s a pricy bit of hardware, but I can leverage it to do all the things one might want to do but avoid for data privacy purposes. I try to avoid oversharing, but when I look at what ChatGPT knows about me, I’m giving way more away than I want to. I look forward to the day I don’t have to use it at all.
Ethically is a slightly different story; there are a number of claims people like to make:
Plagiarism: not in any significant sense. Like any of us might borrow a turn of phrase, so might AI, but any significant body of text — even a paragraph — is very unlikely to match exactly to an ingested work. That’s why they need to ingest so much different source material — so no one author or text dominates the output.
Stealing: Mostly yes. I think there is a significant difference between ingesting all of Reddit (where people including myself were sharing their words with the world) and feeding in copyrighted works of published authors without permission. I think when you turn around and sell access to a model built on material you didn’t pay for, that’s wrong. I do think open models available to everyone are far less evil because they take in information they don’t own and give back a tool they don’t own, having contributed significant money in the form of training.
Infrastructure: Yes but… that’s how everyone does it from Walmart to sports stadiums. Don’t single out data centers when the entire system is corrupt. Companies should pay for transmission lines and capacity they expect to draw. All companies, not just AI.
Environment: Yes but… the amount of water used sounds high on a global basis, but it’s utterly dwarfed by agriculture and some other industries. Now some say, yeah but we have to eat, but do we have to eat almonds specifically? Beef? Do we have to grow food in deserts? Data center water usage looks big in raw numbers, but it’s not in percentage terms. That being said, it can be a huge problem locally where a community might have a sustainable water supply but a big data center tips the scales.
The latter two are IT industry issues that are driven by AI but aren’t going away when and if the bubble pops. Imagine the privacy nightmare that is coming. In fact, I’m not convinced that AI isn’t just a cover for unprecedented data collection.
They are all trained on copyrighted material without permission, no LLM is ethical
As an exercise so I could learn the technology, I trained a model exclusively on the Public Domain works of L Frank Baum, specifically the “Wonderful Wizard of Oz” series (did you know there are 14 books in the series just by Baum?!). The model created is not useful as a tool, and more often than not just produces English gibberish, but every now and then it can produce original coherent statements.
Again, I didn’t do this to produce a useful tool, but rather an exercise to learn how to build models.
They are all trained on copyrighted material without permission, no LLM is ethical
However, this means I can refute your statement because 100% of the input data is public domain novels. Also, I trained it on my own hardware powered 100% by solar power.
Cool that you did that. However, “Large” as in “Large Language Model” generally refers to the dataset size, and 14 books is not Large.
The model created is not useful as a tool, and more often than not just produces English gibberish, but every now and then it can produce original coherent statements.
So we created multitudes of digital byte-sized monkeys…and gave them m/billions of typewriters. . . :p
So we created multitudes of digital byte-sized monkeys…and gave them m/billions of typewriters. . . :p
You’re not far off from describing primative LLMs. In fact, the recent innovation in the last 5 years was the application of a specific technique published in a paper called “Attention Is All You Need” that the secret sauce is “attention” added via a transformer layer. This concept is what made GPT, Claude, etc the life-like responses they have today.
Some don’t speak any “language” they are trained to “speak” and “think” in terms of election orbitals and bonding energy. They are used in pharma and materials science to work on intractable problems like superconductivity and meds for Parkinson’s.
that’s not really an LLM then, is it?
It is, cause it uses the same architecture, https://www.geeksforgeeks.org/artificial-intelligence/exploring-the-technical-architecture-behind-large-language-models/
an example: https://github.com/bowang-lab/scGPT “talks” in RNA and is incredibly accurate and able to do predictions
But it is AI.
true, but that’s hardly relevant as a reply to s comment which is clearly referring to LLMs and not machine learning in general
do you consider “piracy” like zlibrary and annas archive unethical?
If a business is using them, yes.
i agree.
in that case, wouldn’t it be similar ethically if people use these models on their own machine for personal use?
There is Apertus which at least claims to only use permissively licensed sources and respect robots.txt opt-outs. They outline their methodology and source datasets in this document. Though I haven’t checked the actual sources myself.
What’s not ethical is a copyright system thag enforces artificial scarcity where there is no need for it.
Piracy is not stealing, and is not inherently unethical.
Everything is copyrighted regardless of it being owned by a corporation or blogger
I don’t partially care about the former
The absolutely most generous copyright protection length is in Mexico which is 100 years plus the life of the author when created. So lets generously say a total of 160 years. Most of the rest of the world is 120 year max. Anything outside of that is in the Public Domain. Many things had a much shorter path to the Public Domain falling into it in as little as 20 years.
So lots and lots of stuff is not copyrighted.
Human brains are all trained on copyrighted material without permission, no human brain is ethical
Not an expert but I’ve read up a little.
One can pay to license material for LLM’s (Google at. al. are not fans of this ofc.), or simply only use stuff in the public domain or your own internal material.
Private/local models exist for various use cases like not wanting even the chance of your stuff hitting the cloud. Ex. https://ollama.com/
If you have a 4070 or so you can run a smaller model without a issue, which is well within hobbyist range, but larger models quickly balloon hardware requirements to the point where you’re putting in some bucks.
From that perspective local LLMs sound more like classical piracy. Not ethical, not FOSS, but out of the hands of greedy corporates.
they are also doing distillation from the big models from OpenAI & Claude, so open-weight models with similar capability can be available for free and also reduce the big AI companies’ ability to profit from stolen data.
What are the most successful/useful distillations you can run locally?
It’s the backend stuff I wanna figure out so it can convert plain language requests and notes into personal assistant tasks while being flavorfully bitchy about all of it. Apparently RAG and agent tools have something to do with it.
I’ve heard good things about Qwen.
Right… I meant like which specific file.
Qwen.
How much RAM or VRAM do you have? What type of computer/GPU?
But also out of the hands of the small artists/devs they’re pirating.
Its much less targeted piracy than the traditional, and its almost weird to see the rules pirates implicitly respected being broken.
Some of the open models claim not to use pirated content. I don’t necessarily believe it, bit its being claimed.
They call it open-weighted and not open-source because they don’t have the training material (stolen books lol) but still want to pretend they are l33t hackers not bound to BigTech.
Yes, it’s possible to use it locally with a $2000 computer (that’s for the cheapest ones) but you’ll only get a few words per second. It doesn’t matter to the vibe coders who don’t know how to code.
The software to run that is open-source but the training of the model (the “open-weight” black box) requires to destroy the environment at least once.
Last but not least the free models are obviously censored but people don’t care about censorship anymore for some reason. “Tiananmen didn’t happen? Not my problem” without understanding that more is hidden.
Anyway, no, there is nothing ethical about it.
I’ve come round to the idea that I will not use LLMs regardless of how “ethically” they can be sourced because their outputs are harmful in a way that builds up, like a very small dose of poison.
I’m not holding this as a hard stance, just a current analysis of the technology and how it might affect my life in my circumstances. I think we’re well into a world where we need to be deciding if some technologies don’t suit our lives, because there certainly are even more harmful technologies to come.
It might seem like a romantic or artist’s approach, but with so much spiralling out of our control I’m trying to make the active choice to be more human.
In leaving progress to the machines, in letting technology go forward on its own terms and selecting from it, with what seems to us excessive caution, modesty, or restraint, the limited though completely adequate implements of their cultures, is it possible that in thus opting not to move “forward” or not only “forward,” these people did in fact succeed in living in human history, with energy, liberty, and grace?
Always Coming Home - Stone Telling Part 3 by Ursula K. Le Guin
I’ve come round to the idea that I will not use LLMs regardless of how “ethically” they can be sourced because their outputs are harmful in a way that builds up, like a very small dose of poison.
How is LLM output any more harmful than output from a human you don’t know? I would agree with you if one were to simply blindly accept anything an LLM gives you as factual. However, I am skeptical of what humans say too. Some are inaccurate out of carelessness, some out of malice. Critical thinking is the key to protection from both errand LLM output and fallible humans.
Additionally, this thread is about locally running LLMs. One of the most dangerous aspects of most large LLMs is the sycophancy where the LLM will try to tell you what you want to hear, even if it needs to creatively invent things that don’t exist or are not true. If you are not aware, running locally means you have all the controls on the model. You’re not subject to whatever settings a large hyperscaler sets up for you. This means, among other things, you can turn down the “temperature” setting, which is the lever that controls how creative an LLM is. Practically what this means is, if you set it to “0” you are allowing it no creative action. If you ask it a question it doesn’t have actual training on, it will tell you effectively “I don’t know” instead of making something up just to have an answer.
It might seem like a romantic or artist’s approach, but with so much spiralling out of our control I’m trying to make the active choice to be more human.
The older I get the more disappointed I get in humanities large group decisions and actions.
Some companies build out more electrical capacity than they use. Substations are funded by the DC and built to twice the desired capacity - and half is delivered to the community. The generators are run only when utility power goes down, and run on diesel.
However, not all companies are this responsible and the grievously irresponsible ones get the press and make the responsible companies look bad.
Can you name how many data centers have actually built out excess capacity before they started operating?
Most of the ones I’ve heard have simply claimed that they might eventually, which is a corporate way of saying we’ll walk that back once we think no one is paying attention.
“Open weighted” I take to mean that the user can see what sources the LLM/AI was trained upon. Usually this information is a closely guarded corporate secret.
As for “models” this is no different than the different models that a car company offers. Sentra from Nissan, F-150 from Ford, Civic from Honda
If locally run or through a data center, LLM/AIs have two major problems:
- The results can’t really be trusted due to their tendency to hallucinate.
- Frequent LLM/AI usage has been proven to deskill the user as well as a reduction in cognitive ability. I’m dumb enough without getting help from a clanker.
Assuming “run it 100% locally” means on your own hardware with network access, I see no reason this can’t happen given a powerful enough computer.
deleted by creator
I’ll take a crack at it.
I run AI on an old Nvidia p40 purchased online for less than $200. The models I run on it came predominantly from Chinese companies and groups. On balance, they:
-
use much less fossil fuel in producing these LLMs than their Western counterparts
-
produce models that run much better on lower end consumer hardware
As for training data, the inputs used to create these models, it varies greatly but the most popular line, Qwen, Came from the company’s own data from operating such huge networks and systems for so long
I really fail to see how any of that is worse than playing a video game.
None of this takes away from the very real issues around data center build out and Western companies using the systems to scare people and continue an economic bubble. That’s all true and bad. But there’s nuance. Not all AI is created equal
use much less fossil fuel in producing these LLMs than their Western counterparts
Utterly false, given that all the decent Chinese models (especially Qwen) are distilled from Western frontier models.
They literally could not exist without the enormously carbon emissive western models existing first to train them.
Also China has an insanely diversified grid, that utilises almost a global scale of fossil fuels.
So its like green communism at best
distilled from Western frontier models.
What does this mean please (remember the title of the post!) and how do you know?
Basically, it’s really really expensive (from a compute standpoint, and more compute = money and energy) to train a decent LLM just using books and conversations etc. Anthropic, OpenAI, etc spend a truly insane amount of money doing this is in data centers.
Running an LLM like that is also extremely expensive from a compute standpoint, and right now the true frontier models can basically only be run in datacenters.
However, once you have a really good LLM, you can use it to train another LLM. Because it’s basically already refined all its original training data (and is capable of further refining it’d output on the fly), training a new LLM from it is vastly easier and cheaper.
Additionally, you can use a much smaller and more efficient model, so it can run on less powerful hardware.
So all local LLMs that are any good that exist right now, were trained using the outputs of existing super expensive frontier models. They could not exist without the frontier models to train them. This is also how OpenAI and Anthropic create their cheaper models. Mythos/Fable is anthropics current frontier model and they distill it into (use it to train) their cheaper models like Sonnet and Opus.
Wow. as if the original models weren’t rickety enough.
They’re honestly not that rickety. If you’re getting your news from social media (including Lemmy) you may want to actually try them rather than just take the random incidents where things go wrong as broadly representative.
yeah the comment was bad enough I looked at the user. been around for 2 years and no posts or comments till this one. made a note on the account.
Can you prove that? Alibaba (makers of Qwen) is one of the largest data processors in the world. Why do you think they’re dependent on Western systems?
They’ve been explicitly accused of doing so and it would make sense given that they don’t have access to the same level of compute as Anthropic or OpenAI, being subject to the chip embargo and all.
Ok so no proof then?
-
You can run model locally, a gaming PC is enough, may-be not for cuting edge models but if you want to generate clues for a RPG or rephrase a letter it’s good enough. You can look for LM studio and Stability matrix for example
Bonus if you’re using solar power to generate your responses.
It is absolutely possible to run 100% locally, but in practice, at the high end, only quantized models. The full size top end models require hundreds of GB of video ram, and while you can buy that, it’s stupidly expensive. Quantized models can often perform nearly as well with a small fraction of the ram, but they do sacrifice a little in precision.
These models (well the good ones) ultimately all trace their origins to what you’d likely consider “stolen” data. Whether that’s ethical is debatable. If you’re in the “information should be free” camp, there may not be an issue here.
As for the power/environmental impact, for what they do LLMs are actually very low impact per-request. If you’re concerned about your personal AI power use, then I hope you never fly in an airplane, and minimize your driving because those are much bigger issues.
It’s the scale of use that makes AI an environmental problem, and that’s a question about corporate use of AI, not personal use of AI.
Thanks. What are quantized models?
Kind of like “compressed”. It takes longer/more effort to run them, to produce the same result as a the same model that has not been quantized, where that non-quantized version would consume significantly more RAM but produce the result faster. You would typically only run the quantized model when you’re starved for RAM, which most of us are running LLMs locally.
Think like zipping a file with file compression. It takes less space, but has to be unzipped for you to have usable files again.
I’m going to simplify a little here, so don’t take this completely at face value.
Models are, quite literally, long series of numbers (called weights). A model might store the weights in 16 bit numbers (that is, 16 binary digits). The size of the model (and how much memory it needs) is determined by how many weights there are, and how many bits each weight takes. You can take a 16 bit model and rework it to use 8 bit, or even 4 bit numbers. The result intuitively behaves a lot like the same model, but with less precision to the weights. That makes the model take way less space in ram, but also makes it more likely for concepts (encoded in the weights) to overlap, which impacts model quality. Often the effect is that fine distinctions get lost.
As for the power/environmental impact, for what they do LLMs are actually very low impact per-request.
Worth noting that a request is often dozens of requests now that there’s “reasoning,” even for search. As I understand it, a model will take a question , figure out the context (one request), reform the question so it yields better results (another request), if it’s doing a web search there’s requests for each result, another to compare, another to check if it answers the original request, if not it loops and does it all over again. So one request is easily, and often, dozens of requests. This is one way Ai companies can say to investors, “see, look at how much usage has increased.”
Also, we need to factor in the power to scrape and train each of those models, build the datacenters which is near impossible as these companies are not transparent about it and actively try to obstruct investigations into it. Then there’s the redundancy of all these different companies competing and doing roughly the same thing at the same time, as fast as they can, so it’s orders of magnitude inefficient energy consuming before it gets its first user prompt.
Comparing it to other assaults on the environment is not only hard to do, but a case of “the worse negates the bad” fallacy.
I do see what you’re saying. You can account for all of these factors and it still turns out that, largely, individual LLM use just doesn’t use that much power compared to most things people do day to day. Inference is so cheap that even dozens of requests don’t amount to much. I could look up and give you a bunch of numbers, but I don’t think that’s likely to convince anyone who doesn’t do the research themselves. It’s so easy to come up with sources that say what you want. I’d encourage you to actually look into this yourself.
Training costs are higher, but you train once and use repeatedly. Right now, total training costs are stupidly high, but that’s because we’ve got an arms race between the frontier labs to spend as much money and compute as they can for truly marginal gains in quality. The solution to that problem isn’t for individuals to stop using AI, it’s to stop those assholes from wasting so much power.
Individual LLM use is so cheap, that it really isn’t worth wasting people’s energies thinking about limiting that. Instead of being distracted by attempts to make this an issue of personal responsibility, we should be focusing on what will actually make a difference. We should be focused on supporting policies that lead to systemic change. A carbon tax would change corporate behavior right quick, and not just for AI companies.
The power impact was something that in the early days worried me. Looking into it I agree with you to some degree. Like using it instead of a search engine I think is by and large a wash. One prompt will likely take more energy but will give you information that likely would have required searching several times modifying the words and jumping between sites which are rendering all sorts of things. Heck If I booted into a command line and connected to an llm Im almost sure it would be significantly less energy. If you chat for entertainment instead of streaming vidoe also lower energy use. Now I think one thing is in making things. It lets people who otherwise couldn’t make pictures and videos and code. In the large majority of cases what is made is going to be disposed even for folks that eventually make something they care to keep around or use. While using software to do the same uses a lot of energy the only people doing it generally where making long lasting things for projects or such. So that is where I question it. Still I will have it make a picture to use in an rpg or such.
The models can be ran locally, yes. I’m doing it.
As for the ethics of training? It’s not unethical and I’ll debate that all day, if you want to get into it.
No one ever does though, they just rage quit once I get them to a certain point🤷♂️.
Is it ethical for me to steal someone’s car? It was sitting on the street available for anyone to access. Why would it be wrong?
That escalated quickly.
Why is it similar to the theft of a car to you? Please elaborate!
It’s called an analogy. It’s a similar situation morally that you know is wrong. You’re talking something someone else needs without compensating them for it. If you agree this situation is wrong then you should probably also agree the LLM situation is wrong.
I’m aware, but it’s a bad analogy when you dig into the logic. Continue!
If that’s the best argument against it you can make then that’s pretty sad.
Is stealing a car analogous to scraping copyrighted material?
Is stealing a car analogous to pirating media?
Is stealing a car analogous to reading the material and copying it?
What’s the exact angle you’re using here? Please elaborate and then we can continue!
How are LLM’s not plagiarism with extra steps?
careful, you might rage-quit soon
Yeah, real solid arguments; So rage inducing.
The comment right here I’m 99% was sarcasm. I believe they’re poking fun at the original comment you replied to being incredibly stupid.
It’s simply faster iteration of the current creative process, but with a machine. All work is built upon the shoulders of the work before it, regardless of the process.
Creative process?
How isn’t AI this?
At what exact point does plagiarizing a creative work differ from being inspired by a creative work?
Passing off work as ones own?
What classifies work as one’s own?
Work that can be directly contributed to oneself?
In my book, not paying and supporting them financially - > ethical
My issue is when you pay for a subscription, when you buy a gpu specifically for ai, when you buy a model, etc. Because then you are funding the slop machine.
The datacenters would be built next to wind farms if it wasn’t for oil industry bribes. It’s just a scape goat.
Gas turbines need to be outlawed outright and we need proper regulation for all industries. They focus on getting everyone pissed about AI so you don’t start talking about hanging oil execs.
Ellendale ND has a DC in progress. The region has a power surplus. What you are wishing for is already started.
Lmao, price of fuel right now has people going “por que no los dos?”
You can run them locally, yes. There are models that can even run on phones, but usecase is limited. But it can only be considered ethical, if the training data used is listed or ethically sourced IMO.
AI bros on Lemmy will disagree with me, but most open weight models are still trained unethically i.e, theft. Most proponents of LLMs (who I talked to on bsky), who say local models are ethical, don’t fucking use it. They’re larping on socials about how awesome it is, but none of the ones I talked to are using it in their projects. They mess around, realise it is not as good as the “unethical” options, go right back to Claude
Open weight models Qwen, deepseek, mistral, and the Ollama stuff etc are unethical in normal people’s eyes, but “ethical” enough for AI bros.
From what I searched, there are very few that can be considered ethical - Olmo, Apertus, Starcoder(?). But idk anyone who uses these. My friend at IBM said they used Apertus, but it was nowhere near good as ChatGPT, so they no longer use Apertus now. And these models require minimum 6-8 GB VRAM for their lowest parameter model iirc.
Even the open-weight model bros are lobbying to redefine what ‘open-source AI’ means. That should give you a fair idea about people behind open-weight as well
I would unironically argue that a model primarily trained through distillation of closed frontier models, and then released open-weight with an open-source architecture, becomes “ethical” again.
Something something Robin Hood
Rob the poor’s money from the rich and keep it for your community?
Sounds more like feudal warfare than anything, I can’t see any harm to artists being reduced at all
Borderline strawman there but I’ll bite.
Open weight models trained unethically are unethical. Closed weight models trained unethically and then sold back to you for a profit from gas-powered datacenters funded through Ponzi schemes are substantially more unethical.
From there it’s a harm reduction calculus. No, those are never pleasant.
So do you let the closed weight labs conquer the field unopposed just so you can feel better about yourself? That’s a valid stance, FWIW, and it’s also super easy and convenient because you don’t have to do anything. It especially makes sense if you believe it’s still possible that AI will just go away on its own. I don’t, myself, not anymore, so I encourage the use of open weight models, however grudging, so people don’t give money to the closed labs and in the worst case aren’t eventually stuck with the maximally unethical options. And we’ve not even touched on the nightmare labor replacement scenarios that seem every day less unlikely. Am I right? I have no clue. Like you, I’m just trying to make the best choices I can in a world that’s gone to shit. I’d recommend dropping holier-than-thou attitude either way, though, because it doesn’t help our side. Man.
Just as I suspected…
Thanks for taking the time.
there are very few that can be considered ethical - Olmo, Apertus, Starcoder
This is software meant to be run always and completely locally?
Sorry to whine, but so far nobody has eli5’d what “open-weighted” means, or “model” at that… please?
This is software meant to be run always and completely locally?
That’s what is claimed, I haven’t run them locally since I don’t have a good system.
To be honest, I’m not sure if I can eli5 weights and models, but I’ll try. Think of a model like the base - for example, OpenAI has different models like Astra, Sol, etc. These are different models, like different versions of a software or operating system like macOS, but for AI stuff. Like one would download a software, you download a model to perform tasks.
Weights are vales that can influence inputs of these models to get a desired/better result. What most of these models are doing is mostly predicting what might be the next appropriate text/data to the question you asked. When you ask these AI models what 2+2 is, it is not performing a math operation like a normal program, it is looking at its training data to see what the closest option might be. It is doing pattern matching.
These AI models inside can be thought of like an interconnected network, like neurons in our body, that keep passing information to the next neuron and to the brain to make a decision. (Before understanding LLMs it would help to understand Neural Networks first). These AI networks need weights and biases. These networks perform calculations and weights are used to determine how much importance/weight each input can have on the output. Bias on the other hand, is used to shift/change the output so the AI model can ‘learn’ to pattern match better.
What open-weight models, do is they make the model available for download along with the weights. No information is given on training data. Like with ads, ones with most data emerges victorious i.e, has a better model. So these companies do theft, don’t list their training data afraid of getting caught. I forgot which one, but either Deepseek or Qwen (both open-weight) was caught ‘stealing’ from Claude (not open weight). You can probably guess how much these companies value ethics.
I’m not sure if this entire thing goes away, but local models might be the ones left standing when this bubble pops.
Open weight is different to open source. Open Source AI as it stands, the definition requires a model to have entire thing made public - so the weights, biases, training data used, the model. Apertus, Olmo etc are mostly meeting open source AI definition.
If you need to know more, this is what we’d use to refresh our memory before exams :)
I probably might have made mistakes here, English isn’t my first language either. But I hope you get an idea about these terms
If you really need to understand this tech more, I recommend watching ‘AI for Everyone’ course on Coursera from Andrew Ng. It is free to audit, my friends who took his course were hyped (I wasn’t really interested in AI)
