I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
I use tinfoil.sh with locally hosted models, you are giving tinfoil money for the service but they aren’t paying the model trainers for open source models nor are they training models with your data
The LLMs would be of minor use without the ingestion of the Internet to update its weights. It’s a clanker so it has no ideas; you’ve got to feed it whatever you’re purpose is with it.
Depends what you’re wanting to use that AI for. Is it figuring out protein chains that might help make a vaccine? Sure. Is it making media at the expense of creators? No thanks. It’s hard to justify the costs of large AI rollout and honestly, from what I’ve read and heard, it’s hamstrung from the power grid.
Death to clankers.
I donate my compute to https://aihorde.net/ once in a while. got to play around with some cool AI bullshit there. Its fun to play around with.
Its all volunteer run. I got my steam deck pushing small updates to peoples like dnd rollplay kinda stuff. koboldcpp makes it easy to set up and you can use hugging face models or your own.
Theres some really excellent open models out there if you care to look! Most are hyper specialized better than say GPT and such.
Download Ollama. Download one of the smaller Qwen models. Have at it.
Try qwen3.5:9b (a 7.6Gb model) to start.
Friends don’t let friends use Ollama.
Somethin’ happening with them?
I think from a performance standpoint llamacpp is better, and requires approximately the same amount of technical knowledge to operate.
To be honest, I’m not sure how it all works exactly, but I downloaded some models that run locally and i can choose an agent that can run models that will work with my (and your) limited graphics memory. It’s all local and you don’t have any limits or anything and it’s private. No tokens. However, it will run very slowly depending on what you are doing.
I haven’t used it. I tried a couple of times but I was trying to generate an image and apparently it was too much for it and I never tried it again. But maybe that somehow supports rich assholes. I don’t know. I didn’t pay anything and I was under the impression it was open source.
If you want to try, you can go on YouTube and search under something like “running AI locally.” And you should find some tutorials. Good luck.
Personally I have a love/hate relationship with AI. It’s simultaneously amazing and horrifying.
Use it all you want. Don’t listen to people here. They’re fucking stupid. These are the same people who told everyone not to get into bitcoin. Meanwhile all the assholes are millionaires and these people here gave up every space the had to the investors and are still trying to moral police others.
It’s here and isn’t going away. You’re not a hero for not using it. Learn it like any other tool. Get good with it. Use it to better your life unlike people here who will continue to lose power, territory, success and try to drag you down with them.
Simply: don’t be a dick and don’t use AI. Using AI you’re commiting theft.
Such a drama queen. You don’t commit theft.
Don’t use the camera on your phone, it steals your soul!
Good. Nobody should have we’ve capitalized on the internet. It wasn’t a space to sell onlyfans and subscribe to comics about cats. Those people opened the door for every single capitalist who ruined every other space the ever existed. Turned it into a privacy hellscape that wants a picture of your asshole every time to want to sign up to your favorite artist weekly chance to sell you ads.
Fuck them
Burn it all to the ground. Steal everything. Share everything until people trying to capitalize online fuck off.
Use open weight models either small enough to run yourself or via your diverse online host of choice.
without supporting the data centers
online host of choice
I believe there are options to host open models on hardware you don’t own without having to give money to the “evil” data centers. Getting away from data centers entirely is impossible as that is how the internet has worked for a very long time now.
Not to mention that it’s training and not inference where most of the compute is going anyway.
No, and why do you want to rot your brain and/or produce more slop anyway?
Because LLMs now work better for searches than commercial search engines.
Ha! No they don’t! It gives you a hallucinated answer that’s not even on the webpage that it claims it got the answer from.
At producing misinformation.
Why would you want misinformation? It’s going to lower your productivity, because you then have to fact check everything it tells you
I like deepseek v4flash, I use opencode go and commandcode, somehow feels not as bad since its cheaper and technically can be ran at home if you can afford it. I don’t care for regular ai chat, feels useless after using agents. Agents are what you expect when you think of ai at the computer, hermes is a solid free one. Those two subs rotating as one runs out should last you a while at 20$ a month. These smaller companies/datacenters aren’t the issue imo, the larger ones like Anthropic, google, openai, etc. are.
It just functions so much better to use low cost api subs over trying to run anything locally. Let them subsidize this shit for you, don’t abuse your own hardware while it’s expensive af to replace, can spend 100s a year on subs and it’ll be less than the hardware costs for that amount of time.
The size and token rates mentioned in this thread for local inference are all going to be exceptionally limited compared to the frontier
Forget about local inference with your hardware, it’s too far behind. 3 years behind in this technology is basically stone age
What you’re looking for is “ZDR” (zero data retention) policies from online providers. There are many options. One powerful model with ZDR you can use is gpt-6 Luna on duck.ai
Even if you’re trying to code, you’ll get better mileage copy pasting to and from duck than ollama with your hardware
trying to code
copy pasting to and from duck
That’s, uh… The worst way to do any of this. And GPT-6 Luna is the weakest GPT-6, if you’re going to run something that’s on someone else’s hardware anyway, you can just use openrouter and choose ZDR hosters of open-weight models that are somewhere between GPT-6.1 Sol and Terra in accuracy.
OP said they didn’t want to support billionaires and data centers
The money used to make zdr requests on openrouter to good models like GLM 5.3 flows into large inference providers like fireworks and baseten
Yes, copy pasting is a terrible way to vibe code. But my point is it’s still going to be better than trying to ollama with a 1060 and 16 GB RAM
I don’t think I’ve seen this recommended yet, but maybe you can check out CoCore https://www.cocore.dev/. I saw a demo of this some time ago and it’s still pretty much an experimental project. Essentially this is more of a community shared way to run local LLMs, especially for those without necessary compute. You are basically using an LLM hosted on another person’s computer in an encrypted manner. There’s a lot of technical detail about it that goes over my head. It is slow, and of course, still requires trust, or some auditing to verify it. But I think it’s kinda a cool way to crowdsource resource to run. The network of users is on fediverse btw.
Sounds like golem. Interesting
