TrendPulse

tldr: Googles quant is broken, use unsloth UD Q4_K_XL for now This might be low quality post, but oh well, we ball llama-quantize will quant the token embed to q6k when Google really was supposed to use "--pure" but that’s only the first problem The llama-quantize quant function is hardcoded to -7 when SOME groups are actually optimized for 8 The 32 block groups are misaligned which causes them to intermingle, so they just need to be sorted and quantized separately unsloth Q...

MTP for tiny gemmas for mobiles or potatoes or raspberry Pi, or maybe for ants

I've previously posted some small performance benchmarks, but this time I got interested in the qualitative side. u/Substantial_Step_351 posted a few days ago about why models are not benchmarked on tool calling , and u/complexminded pointed out the tool-eval-bench utility by SeraphimSerapis in a comment. This got me interested in benchmarking a few questions that I've wondered about that I don't recall seeing good answers to: Are the ByteShape quants of Qwen3.6-35B-A3B as good as the...

For real tho, these bots need to turn on their web search functions and quit living in the past. It’s bad enough we gotta deal with all the “Qwen3.6 27b helped me quit drinking and brought my dog back from the dead” posts. Sheesh /s

I've been playing with Nex N2 Pro (Qwen 3.5 397B finetune) locally today. I noticed straight away that it has a pattern of reasoning that is distinct and uses simple words like "need" and "maybe" a lot. Here's a sample of reasoning. We need answer user asks "what is the theory of bicameral mind about?" Need explain Julian Jaynes theory. Need concise but enough. Could mention from 1976 book, humans lacked introspective consciousness until ~3k years ago, brain...

https://github.com/pitbox46/NanoQuant TLDR: NanoQuant is a quantization method to create 2 bit/weight, 1 bit/weight, 0.5 bit/weight, etc, quants of dense transformer models. I've followed the paper's methods and created my own implementation which is still very much a work in progress, but currently seems very promising. I am not affiliated with the NanoQuant team What is NanoQuant NanoQuant (Chong et al, 2026, https://arxiv.org/abs/2602.06694 ) is a post-training quantization method w...

I'm not going to. And you shouldn't either. The frontier labs are the ones who are harming our community. They are jacking the hardware prices up. First it was nvidia GPUs. And then it was RAM. And then SSD. And now HDDs prices are x3 compared to last year. Even NAS prices are going through the roof. Really? Don't give them a chance for good exit strategy. Why? The frontier labs are doing this bc they don't know any better way to inflate their valuation. They know that the local, open wei...

They seemed so promising at one point but the biggest ternary model is still 2B. What happened? Why aren't the frontier open weights AI labs attempting to use them?

Since there is much (justified) whining about post quality, I thought it would be helpful to get a sense of what people actually DO like. Here's my take: S-tier: -GGUFs/MLX or benchmark data for new best-in-class local model released - New Optimizations that are actually a big deal for most people (e.g. MTP) - Hardware capability posts that include both prefill and decode t/s and specify engine, quant, and context size. - weird stuff like that robot in the suitcase A Tier: -New opti...

I am making a game that is bundled with a local LLM and every conversation is unique. The game, 'Simulation Simulator', is a campfire chat sim game about DMT, simulation theory, and a friend with a computer monitor for a head. 5 endings you can reach totally based on how you interact naturally with the AI. One is a romance ending! Everything in the clip is totally organic and unscripted. Trying to use AI for good. Haven't seen the use of LLM tech inside games to this extent yet. I'm sure peopl...

Hey fellow Llamas, your time is precious, so I'll keep it short (while trying to explain everything lol). TL;DR: 33-35B MoE on a 16 GB GPU. Qwen3.6 35B-A3B: 13.3 GiB (was ~20.5). Laguna XS.2 33B-A3B: 14.6 GiB (was 18.8). Both measured on an RTX 3090, both under 16 GiB. Only the active experts stay on the GPU. An A3B model routes to ~8 of 256 experts per token. Spark calibrates which experts your traffic hits and keeps those hot; the long tail lives in system RAM and is swapped in o...

Just saw Xiaomi MiMo announce MiMo-V2.5-Pro UltraSpeed , claiming they broke the 1,000 tokens/sec output barrier on a 1 trillion parameter MoE model . According to them, they’re doing it on a single standard 8-GPU node , not custom wafer-scale hardware like Cerebras and not SRAM-heavy hardware like Groq. Crazy if true.

Hello everyone, I've been working on a point and click GUI to make tinkering with llama-server flags much quicker and easier, I thought I'd share for anyone else who might be interested. It's also great for anyone new to llama.cpp that is looking to get into it and doesn't want to deal with the terminal yet. My aim was to make it quite simple and bare-bones so that's why it might look a little bland, didn't really see the point of making it look decent with a nice colour scheme or anything for s...

Hardware: RTX 5090 | Model: Qwen3.6-27B | Framework: BeeLlama.cpp Full benchmark scripts, raw data, config, and generated artifacts are available on request — just DM or comment below. I spent the last week benchmarking DFlash speculative decoding combined with KV cache compression strategies on Qwen3.6-27B. The results are surprising enough that I wanted to share them for anyone running local inference. Setup GPU: NVIDIA RTX 5090 (32GB VRAM) Model: Qwen3.6-27B in two qu...

drop your favorite model and quant in the comments. View Poll

OpenEnv is a tool for creating an agentic execution environment like terminals, browsers, or anything an agent can interact with. And today, we’re excited to announce that OpenEnv is becoming even more open, to make the future of training agents open source. Starting today, OpenEnv will be coordinated by a committee that so far includes Meta-PyTorch, Reflection, Unsloth, Modal, Prime Intellect, Nvidia, Mercor, Fleet AI, and Hugging Face. OpenEnv project is supported and adopted by some of t...

Show your videos to Gemma or Qwen today

These last few weeks have been godsend for 24GB (and below) gpu poor peeps. Killer models released (Gemma 4 / Qwen 3.6) Free intelligence via QAT Bonus speed via MTP We're at the tipping point where GPU poor (24gb and below) people are actually NOT poor any more. I was already happy with Gemma 4 31b running at 40tok/s but now its 70-80tok/s Its not a wonder 3090 prices are increasing. For ref: - limit=1, OSL=192, concurrency 1, temp=1.0/top_k=64/top_p=0.95, ctx=40960, q8_0 KV ...

Improved MTP performance (For Gemma-4) This got merged yesterday. Available b9551 onwards.

I spent the last few days trying to get consistent tool calling out of the new Gemma 4 12b QAT model and had to give up. When the model actually works, it works great, but for my specific use case and workflows it is just not for me. It is a major regression compared to the standard Q5_K_L version, which worked without issue. I know the general consensus is that Qwen is for coding and Gemma is for creatives. But I can tell you for a fact that I code very well with the regular Q5_K_L version. W...

A clinical AI model does not need to know who the patient is to reason clinically. It needs the symptoms, medications, lab results, diagnosis history, and treatment course. The problem is that in real medical records, those facts usually sit next to identifiers: names, record IDs, insurance numbers, addresses, phone numbers, admission dates, department names. So clinical de-identification has a double contract: 1. Do not let patient identifiers leak. 2. Do not destroy the clinical fac...

https://huggingface.co/mindlab-research/Macaron-V1-Preview-749B https://preview.redd.it/g64u0fyts06h1.jpg?width=913&format=pjpg&auto=webp&s=8211f5b58e610b28bed0720e5357269a4519e02f https://preview.redd.it/9vs0geyts06h1.png?width=1035&format=png&auto=webp&s=ba96d1ed4129cdc53f9421f8e41ab38586b1b92b https://macaron.im/mindlab/research/macaron-v1-preview

Watched this with a mixture of disbelief and "this is brilliant/ludicrous" - if you're into offbeat LLM stuff it's worth 10 minutes. Fully Hallucinated Operating System - https://www.youtube.com/watch?v=z3pV6FHvcgM

It’s finally here! Something many of us running dual RTX 3090 rigs have been anticipating. club-3090 has rolled out experimental support for Qwen3.6-27B with FP8 quantization . The official Qwen/Qwen3.6-27B-FP8 model performs virtually identically to the original unquantized BF16. https://github.com/noonghunna/club-3090/blob/master/models/qwen3.6-27b/vllm/compose/dual/fp8/mtp.yml

Hello, I hope everyone is doing well. I'd like to consult with someone who has experience using Kokoro locally, because using the Open Router version, I'm not getting good results in languages other than English. Do you have any tips on this? I want to train Kokoro in Brazilian Portuguese so that he speaks more naturally. I appreciate any tips. Thanks.

Not talking about 31b. In terms of creative tasks, writing, chatting, not necessarily coding but can still be included, Does Gemma 12b outperform in any way? Is the 12b closer to the 31b compared to the 26a4b?

I wanted to try new QATs and opened two collections on HF (which HF found for me): https://huggingface.co/collections/google/gemma-4-qat-q4-0 https://huggingface.co/collections/unsloth/gemma-4-qat One strange thing caught my attention, for e.g. E4B: https://huggingface.co/google/gemma-4-E4B-it-qat-q4_0-gguf/resolve/main/gemma-4-E4B_q4_0-it.gguf 5.15 GB https://huggingface.co/unsloth/gemma-4-E4B-it-qat-GGUF/resolve/main/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf 4.22 GB How can _0 be la...

I'm an old guy and I hate when things change so fast surrounded by noise and breaking news! MTP, I know what the acronym means and where it excels. Gemma4 31b dense is my target. Unsloth, Google, GUFF, tensors... too many overlapped informations. I hate when I see no clear path. Please help me... FACT 1 = MTP has been merged in llama.cpp FACT 2 = old GGUFs are not compatible FACT 3 = I need a second file to load with the GGUF Is fact checking ok? Which GGUF is ok? Why Unsloth add...