I currently run the top 8 or so per day from r/LocalLLama through my RSS reader. Actually going to reddit is eww, so I want it to be worthwhile. I usually get enough to keep roughly up to date just with title and text / graphs and only go if it looks really interesting.

With the upcoming demise of reddit RSS, I’m wondering if people have recommended (hopefully RSS friendly) places roughly equivalent. I guess I could filter HN which may be useful for many tech interests, but I don’t know what I don’t know, so I thought I’d pose the question to the crowd.

  • brucethemoose@lemmy.world
    link
    fedilink
    English
    arrow-up
    0
    ·
    edit-2
    1 month ago

    Actually…

    And I really don’t mean to be pedantic. But Unsloth has enshittified some.

    • Some of their GGUFs target their own vibecoded PRs first, not necessarily mainline llama.cpp.

    • And, more frequently as of recent times, they’re bugged just to get them out the door first, and get the hype.

    • The quantization loss is higher than some other quanters now, again, due to the rush.

    • There are also some much stronger quantization schemes outside mainline GGUFs now, like ik llama.cpp or exl3.

    Unsloth isn’t bad, I’m just saying be careful using them as the defacto quantization source.

    I used to love them. But I haven’t downloaded an Unsloth model in some time, and TBH I am not really inclined to…

      • brucethemoose@lemmy.world
        link
        fedilink
        English
        arrow-up
        0
        ·
        edit-2
        1 month ago

        Yeah, hence the slew of forks.

        Not that I blame llama.cpp. They have a huge job for so little dev manpower, and it’s trying to be a jack of all trades rather than something specialized like other frameworks.

        • pepperfree@sh.itjust.works
          link
          fedilink
          English
          arrow-up
          0
          ·
          21 hours ago

          Llama.cpp is more specialized than huggingface transformers as they are written in specific way(no extensibility) and accept specific format(only gguf). Things like sglang and vllm can do things better than transformers.

          • brucethemoose@lemmy.world
            link
            fedilink
            English
            arrow-up
            0
            ·
            edit-2
            20 hours ago

            By that I meant the targeted scenario.

            Llama.cpp is trying to be a “parallel deployment framework” that optimizes for throughput and works best on huge multi-GPU servers (like vllm/sglang) AND a “local inference engine” that works at a batch size of 1, and optimizes for memory usage on consumer hardware.

            It’s trying to work on everything. AMD, Nvidia, Intel. Vulkan, rocm, CUDA, sycl, AMX, all sorts of exotic frameworks and hardware.

            It’s trying to support Loras, exotic sampling, task vectors, lots of features.

            It’s trying to support every model, from llama 1 all the way to exotic MoEs, as well as embeddings models, decision models, models with audio input/output or image/video input…


            What I meant is that llama.cpp is generalist.

            Frameworks or forks that focus on doing one thing well, (like pure speed on a Strix Halo board, Lora swapping, hybrid inference, extreme quantization compression) tend to have an advantage because they can optimize just for that one thing, and make huge tradeoffs. Their dev manpower is also more focused.

            Hence, while I love mainline llama.cpp, I personally never use it anymore because I get better performance with some hyper specialized fork or framework for specific scenarios. Like sglang for constrained sampling on servers, or exllamav3 to squeeze an MoE onto my Nvidia desktop.

            • pepperfree@sh.itjust.works
              link
              fedilink
              English
              arrow-up
              0
              ·
              13 hours ago

              For me its more like C++ is harder to maintain, llama.cpp dev pick and choose which implementation of same thing for it to be more maintainable. If forks can do it, llama.cpp can do it too. What I mean by slow is their development is slow, is case there is misunderstanding. Python based framework is faster in development because those critical optimization usually written in lower level language. And with wrapper support, ai labs can write the wrapper without waiting it to be accepted to the upstream.