TrendPulse

Edited - After digging into this some more and reviewing unsloth post for better understanding, the divergence APPEARS to stem from I did not use the BF16 QAT model as the "reference" model.... The QAT vs standard Q4 comparison in our benchmark is not apples-to-apples . The QAT models were evaluated against a reference they were never optimised toward. The standard Q4_0 and Q4_K_M comparison is valid. The "QAT is worse" conclusion needs a big asterisk: we can't actually ...

I've been happily using GitHub Copilot for 7-8 months, primarily in Visual Studio and VS Code, mostly with the built-in flagship models and have felt like the output is worth the cost. Lately I've been playing with a lot of different local LLM models and decided to try using Qwen3.6 35B on some non trivial programming tasks within a repository that is 10ish years old with thousands of files and multiple languages in it. I've actually been really impressed when I use the "ask" functiona...

I just told it, make a minecraft server and let me play and it worked lol. I just asked "host a minecraft server so I can play" and it did host it, made me a dashboard ands its crazyyyyy lol, It is hosted in hongkong somewere TwT

Im into both HPC and 3D reconstruction, so I built this as a side project. dvlt.cu is a single 5MB binary: - No python, torch, TF, ONNX, llama.cpp, vLLM, or huggingface runtime - Nearly no dependencies: only cuBLASLt (shipped with libcuda ) + cuTLASS ( header only lib ) - mmap'd bf16 weights, one bulk GPU upload, static dims, one-shot arena, deterministic - Weights (117M Params) are NVIDIA's (non-commercial), fetched separately at setup. - Just download the weights, build, and ...

First we never saw an upgraded Air model after 4.5. Then GLM 4.7 Turbo was great, but quickly surpassed for coding. Now GLM 5.1 is a coding beast, but too huge for most to run locally, and even slow on API. Will we ever get another Air model with frontier reasoning and knowledge? Or a turbo model that surpasses Qwen 3.6 35B in agentic coding with way fewer tokens? Will you QAT like Gemma to leave Qwen in the dust?

Couple of years ago, before Local LLMs were in vogue, I bought 8 RTX 3090 @ $700 each to build a AI rig, it been working great and I was looking to build another to increase my capacity but looking at EBay those are now selling for 1,300 -1,500 range! That price seems totally crazy because on my main machine I have 3090 Ti that I bought new 5 years ago for about 1,400. Needless to say, I was in shock and started looking for other GPUs. Then I went to Amazon and can buy a brand spanking new 3...

It's an incredible little package but too expensive of a price to pay for the performance and I simply didn't want to be part of the great "Superchip lie" - it could be super, but its super ruined by its limited memory bandwidth even though it *could* be 2x throughput - it isn't. (The c2c is 600gb/sec but the memory isn't) I wanted better experience for larger models and these fail miserably at 27b and do ok at MoE's which actually shine on much cheaper hardware, and I don't have the...

There is probably some relevant information to other cards here but my benchmarks are on dual MI50 32GB cards because that is what I have, and thought I would share with the community. Install instructions at the end. I'll put a dump of the full llama-benchy tables in a comment in case anyone wants them, they include 3,4, and 8 concurrency levels, too (edit: it won't post the comment, maybe because my internet sucks, I'll try to later). For those that don't know, llama.cpp is available in the ...

I have a 5090 power limited to 475W. When I run the following command, it barely hits 300W and I get something like 30 t/s: bash ./llama-server \ -m ~/myp/models/unsloth_mtp_Qwen3.6-27B-UD-Q5_K_XL.gguf \ --host 0.0.0.0 \ --port 8080 \ --chat-template-kwargs '{"preserve_thinking": true}' \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ -fit on \ -c 131072 \ -fitt 3000 \ --spec-type draft-mtp \ --spec-draft-n-max 2 \ -n -1 \ -fa on \ --repeat-penalty 1...

I know this gets asked a lot, but I can only find threads that are at least a couple of months old, so I thought I'd ask to see what people are running these days. I have an RTX5080 and 64Gb Ddr5 RAM. What's the best I can run for coding? And for agentic workflows? If you have a similar setup I'd love to know what quants you are running of which models, and a llama.cpp command with your settings would be sweet too :) [link] [comm...

Table from https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis I heard that MoE models are usually more susceptible to quantization error, but what happened with the 12B? I thought lower-parameter models usually quantized worse and yet, E2B/E4B are pretty much perfect while the 12B deviates from FP16 the most. Do we have an explanation for that, or did maybe something go wrong during quantization-aware training on Google's side with the 12B in particular? I'd also be interested in the e...

I’m a web developer doing mostly coding, but also project management, requirements analysis, testing, etc. I recently started experimenting with local LLMs, mostly because agentic stuff finally made them feel useful. Note: This text was fed to chartgpt to fix my messy repeating grammar My initial impression was honestly pretty discouraging. Endless model option confusion, benchmarks that are hard to translate, huge VRAM requirements and hardware prices that are completely unreasonable. Still...

Now someone needs to quantize them to 4bit, also I have intentionally kept the divergence and refusal different from original Gemma 4 heretic collection, so you can even try these as alternative to original model.

Google just released the QAT (Quantization-Aware Training) variant of their Gemma 4 models, including 12B, so it was only natural for me to benchmark it on my 12GB GPU since it fits entirely in VRAM. I was pleasantly surprised with the result! By using llama.cpp patched with the Gemma 4 MTP PR, and loading Unsloth's gemma-4-12B-it-qat-GGUF quant and Google's gemma-4-12B-it-qat-q4_0-unquantized-assistant QAT assistant / draft model, which I converted to GGUF and uploaded to HuggingFace as ...

TL;DR Based on long context KLD benchmarks, KVarN appears to be just better than usual llama.cpp KV cache quants. At every size, KVarN matches precision of usual quants of one bit higher. A number of people in the comments under my previous post asked a fair question: what if we drop the obsession with 2-bit and 3-bit toy quants and apply KVarN to high end? So I did just that in my latest BeeLlama v0.3.2 Preview (fork of llama.cpp with DFlash, in short) and ran the same benchmarks a...

Hey, Nick here from Cohere. Thanks for all the feedback on Command A+ the other week everyone. I read these threads all the time about other releases so it was fun to read one about our own :) we would like to do more of it. We actually have our first coding model we’re getting ready to release soon, and I wanted to give this community an opportunity to test it out and give feedback before we officially release it. Figured why not try something different and get you guys to help directly her...

Hey r/LocalLLaMA , Before anyone jumps on me, this is absolutely not a post about how great Qwen is 😄 Even though I use Qwen 3.6 35B-A3B daily, I’ve found a massive gap between Opus and every other model, local or frontier (including GPT 5), when it comes to low-level systems engineering. I’m using that phrase deliberately to avoid just calling it "hacking" 😄 I’ve spent the last few months burning through a serious amount of Opus tokens on this project: https://github.com/miha...

This is the StepFun Step-3.7-Flash UD-IQ4_XS main model with the official StepFun MTP Q8_0 draft model, served through a patched llama.cpp Vulkan/RADV build. Host System: AMD Ryzen AI Max+ 395 / Radeon 8060S ( gfx1151 ) Memory: 128 GB unified LPDDR5X BIOS UMA / VRAM: 4 GB UMA dedicated VRAM GTT ceiling: 112 GiB IOMMU: enabled ( amd_iommu=on ) OS: Ubuntu 25.04 (Plucky) Kernel: 6.18.1-061801-generic Mesa / RADV: Mesa 25.2.8 / RADV ROCm: 7.1.1 baseline; some later ro...

Gemma 4 QAT Q4_0 Bench on Strix Halo These are Google's official Gemma 4 QAT Q4_0 GGUF models, served locally through llama.cpp Vulkan/RADV on a Strix Halo APU. QAT means quantization-aware training . Instead of taking a normal model and quantizing it only after training, the model is trained or adapted while accounting for the lower-precision format it will run in. The goal is to make a small Q4 model keep more of the original model's behavior than a simple post-training quantization. Ho...

I recall in earlier days, q4 was said to be optimal. That is to say, if you have a: small q8 model medium q4 model large q2 Assuming they use the same amount of GPU VRAM, medium q4 would be the best-performing model. I also know that Apple (crazy that I am citing Apple here, given how secretive they tend to be) was quite public about using q4 quant models for thier on device.

Up to 5.8x throughput speedup on Qwen3 Paper : https://arxiv.org/abs/2605.29707 Code : https://github.com/jianuo-huang/Domino Models : https://huggingface.co/Huang2020

I've been tinkering around with LLM for a while now, started with LM Studio like probably all of us and wanted to go into headless selhosted model so that I can use my macbook and still use my AI models. I've been using Qwen 3.6 (and 3.5) 27B on my main computer which has a Ryzen 7 3800X, a 7900XT, 32Gb of RAM and that thing was pretty sloooooow even with MTP enabled. You can probably call this a skill issue as I'm not familiar with llama.cpp forest of arguments yet despite reading the docum...

First time hearing it. I also heard about the gemma 4 qat quants and if any one of them is good for 4gb vram and 16gb ram. I can run gemma 4 26b moe iq2 nl at 8.5 to 9 tps(kv cache unquantized on gpu) with 9 layers offloaded to gpu

TL;DR: I spent a long session tuning a 35B MoE on a tiny 8GB laptop GPU. Three things mattered a lot (--no-mmap, VRAM headroom, closing CPU-hungry apps). Several "obvious" optimizations did nothing because of this model's hybrid architecture (TurboQuant, Flash Attention, even i-quants made it worse). And speculative decoding gave me +26%, which contradicts the community benchmarks that found it net-negative. Looking for discussion + ideas. The setup - GPU: RTX 4060 Laptop, 8GB VR...

Model Summary: Granite Vision 4.1 4B is a vision-language model (VLM) that delivers frontier-level performance on structured document extraction tasks — chart extraction, table extraction, and semantic key-value pair extraction — in a compact 4B parameter footprint, providing a lightweight alternative to much larger frontier models for these tasks: Chart extraction: Converting charts into structured, machine-readable formats (Chart2CSV, Chart2Summary, and Chart2Code) Table extraction: ...

Here model: https://huggingface.co/LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Claude-4.6-Genesis-APEX-GGUF New features: Stability for coding. Even on Q4_K_M quant (APEX Compact), with complex roleplay System Prompt. Short thinking chain. Both thinking and non-thinking modes work. I prefer thinking mode. Fully uncensored with Claude 4.6 Opus reasoning. Improved function and tool calling. Model is based on this release and made via delta merge: https://www.reddit.com/r/LocalLL...

In case you're not aware already, the DeepSeek V4 series is finally getting supported on llama.cpp with this PR ! The PR is at a very early stage right now, so only try it if you're consciously willing to experiment out of curiosity and accept severe stability/performance tradeoffs. It runs very slow (5-6 tps), GPU and FA support need work, etc., but it is reliable-enough already for correctness. This is my most anticipated model and I had some time to spare, so I ended up downloading the H...