Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
You must log in or register to comment.
I don’t want to create an extra post, so I ask here.
Where do you guys get your hardware from, for an affordable price?
I’d like to run at least a 27b model but my hardware is too weak. I’m on a 16GB unified mem, M4. Tried running Bonsai 27b (it’s Qwen architecture) but I had to close all other apps and it is not in a state that I could work with it.
A Tesla V100 is fairly affordable but older and won’t run newer quants like FP8 or attention kernel routines. But it will run a pile of models, even most new models are compatible or usable.
I wouldn’t mind running a model at home, mostly to help document my setup. But I just don’t trust it. So no.
Why wouldn’t you trust a local model? I mean sure they are less intelligent but if you dont expose it publicly you should be good to go
I don’t trust it to be accurate; I’d have to reference the sources it pulled from. Extra work for little payoff.
Got it, makes sense. You’d probably have to do some benchmarking first. Depending on how large the source catalog is.
It depends on your hardware. But right now the “cheapest” hardware you can buy for a usable setup, is the strix halo (128GB), and run gufo with qwen 3.8 flash next (Q4XL). It is sonnet 5 level. Very good speed (both prefill 1000+ and decode 50+), with up to 256k context.
I have it set up with omp + searxng. Sometimes with hermes. Really good! It changes the game for local setup.
strix halo (128GB)
googles
Jesus christ
Just get the cheapest OEM, Bosgame M5. I know “cheapest” is in quotes. But again, it is the cheapest 128GB unified vram device you can get right now. I got mine at pre-orders, so yes, more than a year and a half ago, prices were a steal. Also, great desktop (mini) PC overall and very efficient, even if you don’t want to do AI.
Qwen 3.8 flash next on a 4090, using strata instead of llama.cpp, gives me about 45-55 tokens/sec. Mainly used for coding tasks. The model size is perfect. If the qwen 4.0 models can be a bit less wordy in reasoning but the same size and quality they’ll be really strong.
24GB GPU
Qwen 3.8 ( 2 variants )
Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab
Zoo Code>OpenWebUI>lama.cpp for coding , mostly scripts and html
Qwen 3.8 was a game changer for me.
What variants? And why two?
For coding with large context: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp Anything else: Qwen3.8-27B-GGUF:Q5_K_M
As a service? No, not really.
I use llms for specific tasks, usually small models, for things like identification and tagging (immich, paperless), translation or transcription (whisper), etc. I have a bunch of larger models on a Mac mini and my work laptop for various testing purposes, and have used them to make summaries and other such things.
They aren’t great for search IMO, and search summaries can be quite bad.
I do find them useful to interrogate an unknown rs232 protocol or similar, though, which is where larger models come in handy.
Corporate LLMs are generally garbage to me not only because they are just data farming, but also because they really aren’t good at everything. They might claim it, but the reality is a tailored model is often the better approach.
Using openwebui as a front end for llama.cpp is a decent strategy, combined with sub-agent routing (so generic model hands off to tailored models then returns the right info). This can be useful for some scenarios.
A caveat on coding - IMO, they are mostly bad at it. If you are using it to generate a small part of something that needs to be repeated they are handy, but at scale they aren’t great, meaning for the entire project (or even an entire plugin or module, depending on complexity). Maintainability becomes a serious issue. While they have improved somewhat, there are limits to what they can do.
Macs with M series chips are good options for a hardware base since the memory is shared, and IMO is one of the most cost effective options over a dgpu with a ton of memory. A dgpu with a decent bit of memory (say 8-12GB) is a solid choice for a model thats larger (27b with 3 or 4b active as a mixture-of-experts model) especially if you have an iGPU for regular graphics tasks. Since its all shared on the M series macs, just get something that fits entirely in memory there.
If you haven’t seen it already, take a look at [email protected]
My server’s old GPU (8gb) can’t run anything useful LLW wise I don’t think.
Have run some stuff locally on my main computer, but nothing setup as a proper service.
Not for complex tasks, really. Gemma4b or similar might be just enough as the backend for a voice assistant replacing Alexa (which I honestly hope, I need to get family members off of that fucking spy devices…). Small web searches might just fit. Or image recognition for Nextcloud Memories, stuff like that.
Immich’s image recognition works fine off the CPU. I did get it running an LMM, but it was useless.
I self host both at home and at work.
At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I’m trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone’s life a bit less admin-y without having to hire a full PA role.
Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC
Are you planning on using a specific package for your PA agent? Or is this something you plan on developing in house?
Using openclaw for the harness and then attaching office 365 mcp with an Entra ID App token to give it access to email, calendar, and teams. The MCP format means that we can explicitly control what it reads and writes
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters NAS Network-Attached Storage VPN Virtual Private Network VPS Virtual Private Server (opposed to shared hosting) nginx Popular HTTP server
4 acronyms in this thread; the most compressed thread commented on today has 14 acronyms.
[Thread #135 for this comm, first seen 17th Oct 2026, 13:50] [FAQ] [Full list] [Contact] [Source code]
Not me. Call me old-fashioned, but I prefer organic.
I’ve thought about bodging something together with n8n and Tasker so I can just screenshot something or send it a voice note and have it add an event on my calendar.
That being said my server is an 8th gen i3 w/ 12GB of RAM and I’m not willing to leave my desktop running 24/7 so idk if I can run a model good enough to handle this over there. Even if it is a supposedly menial “OCR and call tool” workflow.
Tried running a decent Qwen on a 3060 TI but I only have 16g ram so it kept getting stuck and not completing tasks.
Ollama container, 2070 super, 3-7b, meta and qwen
I use it to ingest tons of news rss, find dupes, summarize, and then post that output to mesh a mesh network.
I also use it for a self hosted mealie to figure out recipe->ingredient.
I also use it for receipts. I translate pdf -> txt, feed the text to qwen, ask it who is the vendor, what is the price, what it a reasonable memo for the receipt, and give a dozen categories, which is best.
I am actually very interested in the news feed thing. How do you find duplicates in the news feeds? Wouldn’t you have to pull the actual articles and then have MB of data to shift through which would exceed the window? Do you use a vector DB? Do you just have it use curl or do you pay for an Internet API service?
Pardon the mess, this turned out to be quite a lot of text.
No vector DB, no paid API, no full articles. Plain Python pulling free RSS feeds, plus Ollama running on my own 2070 Super. RSS already gives you the headline and a short blurb, so there are never megabytes to sift through. Every Ollama call is one story at a time, so the context window isn’t an issue. Llama3.1:8b for summaries and scoring. Gemma4:e4b for clickbate work, nomic-embed-text for headline duplicate vectors.
Find duplicates:
Python: strip each headline down to keywords (drop words like “the”, “says”, “update”) and only compare headlines that share an uncommon word. Two headlines match if they share enough keywords.
Python: sends batches of 128 headlines to Ollama
Ollama: turns each headline into a vector using nomic-embed-text
Python: compares the vectors, and anything over the config threshold counts as the same story. The vectors just live in memory for the run, no database needed. If Ollama is down, the keyword match still works on its own.
Regen short summaries:
Ollama: Generate a 2 sentence summary from the feed text if available
Get rid of clickbait:
Ollama: Detect clickbait from the headline only. Classify this news headline. “informative”: it states the news itself (who did what, what happened, what was found). Dramatic or emotional wording is fine if the fact is stated. “teaser”: it withholds the key fact to make you click: e.g. “The Surprising Reason…”, “The Class That…”, “Why X…”, “How X became…”, “What we know about…”, “Is X about to…?”, “Then things got weird.”
Unwrap clickbait and rewrite the headline if possible, otherwise drop the story
Ollama: A clickbait headline hides a key fact. Question it leaves open: {withheld}
Does the text below explicitly answer that exact question? Mentioning the topi talking around it, or answering a different question does not count.
“answer”: the answer quoted or closely paraphrased from the text, or “” if the text doesn’t give it. “headline”: if answered, a plain, factual replacement headline (at most 90 cha states that answer, using only facts stated in the text. Keep the text’s certainty: …
Respond only with JSON: {“answer”: “”, “headline”: “”}
Headline: {title} Text: {text}
Score each headline
Ollama: return a summary and an importance score in the same call:
SCALE = ("1-3 niche, local or trivial; 4-6 notable; 7-8 major national or inte “9-10 historic events or emergencies affecting millions right now.”)
Python: rank = importance + a bonus for more outlets carrying it - points for age
Don’t put US articles in World unless they have non-US countries involved
Ollama: Which countries is this news story about? List the country where it takes place and every other country that is a real party to it (acting, targeted, negotiating, fighting, trading, sanctioned). Use common English country names, e.g. “United States”, “United Kingdom”. Ignore countries only mentioned in passing or as background. Always name at least one country: if the text doesn’t say wh give the country where it most likely takes place.
Respond only with JSON: {“countries”: [“…”]}
Headline: {title} Text: {text}
Python: if the only country is the US, it’s left out of World
Everything gets cached per story, so each one only goes through Ollama once.
Wow, this is an absolutely wonderful write-up, thank you!
Hardware:
2 x ASUS GX10 128GB(each)
2 x Mac Mini M4 Pro 24GB(each)
The GX10s are clustered and run DeepSeek-V4-Flash-0731-Abliterated. The M4 Pros run various utility LLMs such as Qwen 3.8
Software:
OpenWebUI
Hermes Agent
LiteLLM
Llama.cpp
vLLM
StoryBinder - custom web app that I use for story/script writing when in the mood
LASM - Local AI Stack Manger - custom web interface to manage pretty much all aspects of my local stack. Agents run on the 4 nodes as well as my NAS(Download GGUF/HF snapshots once and copy out to the nodes)
Have yet to connect my Home Assistant into the stack.
For mobile access I use Conduit https://github.com/cogwheel0/conduit
- 96gb ddr5
- intel i7 12th gen
- RTX 3060
it also doubles as my media ripper. I initially built it specifically for that before AI was even a thing.
I run too many models to count really. everything through CLI because I’m dissatisfied with all the sloppy UIs.
96gb ddr5
Look at the billionaire flaunting their wealth
in my defense I bought the majority of that in 2024 for around $300.
I was surprised to see it’s currently listed at over $2000.
