TrendPulse

I know it might be a no-brainer in retrospect, but hear me out, y'all, it's not the whole story. [tinfoil-hat] What is the hidden strategic value of Gemma4-12B beyond the stated "laptop friendly" size? Looking at the new architecture one can't help but notice that the potential quality tradeoff of an already small model might be too brutal - all your parameters are now doing work on heterogenous inputs. In the latest benchmarks it appears that Qwen3.5-9B is routinely outperform...

Saw this on other sub so posting here. For Intel ARC card holders. Big boost so update llama.cpp version( b9519 onwards)

I picked models I consider local (usable on 3×3090), so there are no 300B models, and you should probably skip 200B models too (but MiniMax and Step are pretty fast in Q3) Gemma-4 12B is still missing

No numbers. Not sure if anybody cares… I’ve run the UD version of Q4_k_m for a month. I talk to this model nicely, because it’s a functional nervous wreck. And initially I thought that might be an alignment thing, so I also have the heretic version when I need a breather from this hyper vigilant over achiever llm. Don’t get me wrong. It’s great ! Works well .. most of the time. It’s when the context gets long(in my case 20k!), chain of tools gets long , or it knows that it previous has made...

https://preview.redd.it/082gnmin1l5h1.png?width=1740&format=png&auto=webp&s=2c89f6310c8c654611188183de07857d77cb2417 https://preview.redd.it/169tjrzn1l5h1.png?width=710&format=png&auto=webp&s=9a1fa656ea95037622b0d7ea2e16a23d2122442c I just noticed

impressive speed/performance ratio! served by togetherAI :)

I completed a Python bug hunting benchmark with Gemma 4 12B. I used the Unsloth Dynamic Q5 GGUF model. The model has good capabilities. Default settings in LM Studio disable the reasoning. Fix the LM Studio reasoning configuration. LM Studio looks for Qwen tokens. Gemma 4 uses different tokens. Change your settings with these steps. • Open your inference settings. • Add this text to the first line of your Jinja template: {%- set enable_thinking = true %} • Set the start token to <|ch...

Curious to know what quantization level (GGUF/EXL2) you find balances speed and smarts for daily use.

I’ve been doing lots of testing back and forth with this 7900xtx. All of my workloads were relying on qwen3.6 models, which are amazing fwiw, but I wanted some diversity in thought. Namely for Honcho workload tiers and differing cron jobs. Not every workload benefits from an agentic-tuned model, so I’ve been testing out Gemma 4 models more. They also dropped quantization-aware training versions of the Gemma 4 family, which reportedly maintain the fidelity of BF16 weights, but with Q4 weights. ...

🔗 Blog: https://rednote-hilab.github.io/dots.tts-demo/ 🔗 GitHub: https://github.com/rednote-hilab/dots.tts 🔗 Technical Report: https://arxiv.org/abs/2608.16894 dots.tts 🎙️ New open-source TTS from RedNote (Xiaohongshu) ✨ 2B parameters (Apache 2.0) ✨ Fully continuous architecture (no codec tokens) ✨ 48 kHz synthesis ✨ Zero-shot voice cloning ✨ Direct text → speech (no phoneme pipeline)

Hi locallama community! Yes, I know, yet another AI agent announcement post. There are a dime a dozen out there... most of them though, are vibecoded, often very sloppy, and eat through context like no tomorrow. This is different. This runs beautifully and very fast with local models on modest hardware. I've spent months working on this in my free time, with lots of manual coding, and i use it as a daily driver in my personal life, as my personal assistant managing my calendar, todos, that kinda...

Their collection: https://huggingface.co/collections/unsloth/gemma-4-qat And their guide, always a very interesting read: https://unsloth.ai/docs/models/gemma-4/qat

This is a PSA for people like me who tried it and hit the wall with tool calls failing left and right, so much so that harnesses like OpenCode just didn't work: There is a fix for that. You need to pass a better chat template file, which is available (I did not write it). See also this comment. To actually use it with llama.cpp, first compile llama.cpp from source, then download the chat template file I linked above, then try this (8 bit quant in this case): ./build/bin/llama-server...

So, llama.cpp has the -nkvo ( --no-kv-offload ) option to offload KV cache to RAM instead of VRAM. Many people avoid this because obviously it hurts performance. But every option exists with a trade off. And in my case, I think it's worth it. Hear me out. I'm running Qwen3.6 27B (IQ4_XS) on RTX 5060 Ti 16GB and 32GB DDR5. In order to fit 65k context, I have to quantize the KV cache down to q4_0, and keep only 58 layers on the GPU. This gives me 23 tps at peak, down to 16 tps during long g...

Of course I’m thankful for all that Qwen has bequeathed us, but deep down in the darkest pit of our souls, every last one of us are just all sitting here waiting for Qwen to say “Hey Google, hold my beer while I drop the best GD model of all time on these fools” /s

Google's collections: https://huggingface.co/collections/google/gemma-4-qat-q4-0 https://huggingface.co/collections/google/gemma-4-qat-mobile And Unsloth's: https://huggingface.co/collections/unsloth/gemma-4-qat Unsloth's analysis (KLD and such): https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

RTX Pro 4500 Blackwell About one month ago I asked the fine people of Reddit for some upgrade advice, on where to take the following AI server next. AMD Ryzen 7 7700 CPU ​Corsair Vengeance RGB DDR5 5600MHz 32GB (2x16) ​RTX 5060 Ti 16GB At first I was considering upgrading system RAM to 96GB to enable larger MoE models, however the feedback was clearly in the direction of "VRAM is king no matter what" and to be honest, there's not much happening around model sizes in the 100B ...

See this question at least a handful of times when browsing new and in the comments, llamacpp has one of the cleaner model hotswap apis now that just works with openwebui and hermes. Bonus: the 2nd model gemma went derp as i was recording this, but the time spent swapping has gotten stupid fast... I remember starting a load and talking a walk while pytorch did its thing just a few months back podman run -d \ --name llama-qwen36-router \ --device nvidia.com/gpu=all \ -v /data/models:/root/.c...

It appears like Unsloth pushed MTP GGUF weights (Q8, F16, BF16) for 31B, 26B-A4B, 12B. https://huggingface.co/unsloth/gemma-4-31B-it-GGUF/tree/main/MTP https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/tree/main/MTP https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/tree/main/MTP

The amount of fast ram is the single most important factor for llm use. There are lots of people that run setups with massive amounts of ram. Reading a post about how model X performs, it'd really help to know the kind of setup being used, otherwise its not relevant for a lot of people. It will also allow easy filtering of posts relevant to the hardware you have, right now thats very hard to do.

Saw this post here yesterday: KVarN: new KV-cache quant from Huawei. 3–5× KV cache compression with actual speed-up instead of slow-down, and unlike TurboQuant it holds up on reasoning (Apache 2.0, vLLM single flag) Cheap KV cache with good precision? Sign me up! Oh, vLLM only... Wait, I do have my own llama.cpp fork , and I do have an extensive reference for KLD benchmarking . I should act! And so I acted. Until 6 am. So now KVarN is implemented in a publicly available BeeLlama....

If you’re building LLM apps and feel confused about when to use keyword search, embeddings, rerankers, or vector databases, this repo is for that. I built a docs-first repo on practical LLM system design patterns, covering pre-filtering, hybrid retrieval, rerankers, in-memory scoring vs vector DBs, batching, cleanup, and LLM-as-judge evaluation, with simple Python examples. From my experience, embedding quality or RAG alone is rarely the full answer. The engineering harness around the LLM us...

Build and play AI musical instruments on your laptop!

SupraLabs just released a new model! - Supra-50M-Reasoning Hello again r/LocalLLaMA ! Supra-50M-Reasoning (ThinkSupra-50M) is the reasoning version of Supra-50M-Instruct. It produces a full thinking chain before every answer, fine-tuned from Supra-50M-Base using a custom synthetic dataset of 500 samples generated by Qwen3 1.7B, trained for 6 epochs. It's experimental, it hallucinates, and it's fully open. This is part of the Supra-50M collection under Project Chimera. Model: 🤗 Supra-50M-Re...

Using `--spec-draft-type-k q4_0 --spec-draft-type-v q4_0` might actually decrease your context size! With quantized spec draft, my context size is 83200. Without it (i.e. using the default fp16 spec draft), context size increased to 91648. I reported this in a llama.cpp discussion and am17an (the GOAT behind MTP in llama.cpp) confirmed my findings as expected: https://github.com/ggml-org/llama.cpp/discussions/24102 Edit: I am using a 3090 for inference. This might or might not apply to...

We are happy to share cyankiwi AWQ update: better AWQ implementation, now with NVFP4 and FP8 Dynamic quantization support. We measured KL divergence against the BF16 baseline for 4-bit Qwen3.6 quants, on synthesized Qwen3.6 BF16 GPQA Diamond responses. cyankiwi AWQ release comes out lowest on both the 27B dense and the 35B-A3B MoE. Qwen3.6-27B (dense) Model Weight size KLD Lorbus/Qwen3.6-27B-int4-AutoRound 17.69 GiB 0.031682 Intel/Qwen3.6-27B-int4-AutoRound 17...

Is anyone still using GPT-OSS-120B? How has it been for tool calling, summarization, coding assistance, and other relatively simple agent tasks? How does it compare to newer open-weight models like Gemma 4 27B-A4B, Qwen 3, DeepSeek, etc.. ? I’m particularly interested in reliability, instruction following, latency, and overall cost/performance. I’d love to hear real world experiences.