TrendPulse

Ran DeepSeek V4-Flash-0731 — the full official checkpoint, not a re-quant — on commodity used hardware. Sharing because I couldn't find anyone else publishing Ampere results for this engine. Edit / update: a commenter called out that hybrid CPU-GPU posts always publish decode and never prefill. Fair hit — I didn't have it. I do now, it's in a new section below, and it's the number that decides what this box is actually good for. Why bother with a 2018 server The model is 156 GB. That num...

Qwen announced Qwen3.8 a few hours ago, and it looks like we’re getting a new 27B model! Really excited to try this one locally.

https://github.com/zai-org/z-ai-sdk-java/commits/glm-5.3

Qwen3.8-Max (2.4T) is another massive contribution to the open weight community. On benchmarks, it performs closely to Kimi K3 and DeepSeek V4 flash across all categories and is better at coding and software tasks. Qwen3.8-27B will also be open weight soon too. Weights are being released next week. Pricing: Input: $2.0 / M tokens Output: $6.0 / M tokens Implicit Caching: $0.25 / M tokens

Every time a model drops from a Chinese lab the thread fills with people who already know who made it, and the guess is usually Alibaba. There was a thread here recently asking what separates the open source labs from the frontier labs. It ran to nearly sixty comments and hardly anyone in it separated out the labs on the open source side. They aren't one bloc and haven't been for a while. I work on the Ling models at Ant, so I'm one of the ones getting lumped in. Discount the paragraph about m...

So this is the stuff of absolute insanity. In less than 20 months we've gone from super expensive cloud models only, to being able to run a Q3 quant of DeepSeek on an Intel Windows PC with a very average 24GB of VRAM. No wonder the big boys are panicking (and yes it's slow as porridge). https://ibb.co/zTvqR8YR

Super excited about this release for the new 27B. Who else is with me. Only 17GB VRAM needed 😍😍

Hello Can you please share the tempature you get on your rtx 3090 under active llm load? Im trying to findout if my rtx 3090's tempatures are healthy or not please share VRAM Tempature only, you can track it via gpu-z on windows

Has anyone yet tried to extract experts from kimi (or GLM 5.2) per chance? There is REAP that removes experts based on routing, but I could only run one K3 expert on my hardware. Could there maybe be a usual expert for different tasks? Would be fun to have a 104B Kimi model, although I think that it would be garbage

I am trying to figure out if it's worth upgrading my RAM, but I've noticed that some MoE models don't seem to do well with many layers shared from VRAM --> CPU/RAM. This may be something on my end; a software config or perhaps my specific hardware config. This made me curious as to how many are doing this. I'm thinking many are, especially with non-dense models, but even better if you do this with dense models; I'd like to know the results you get! Example: You have 16gb of VRAM but you h...

Hey localllama brochacos, what's up? I'm u/raydestar , long time local llm fan. SWE with about 10 years xp, and I have been cranking hard trying to skill up with agentic AI recently. Since open weights got good, it's just blown my mind. What's been hard to understand is "Why isn't this a bigger deal?" I think the answer there is, it's just not as accessible, and most people think that an 8B model just isn't good enough to do most tasks. And -- drop the model in LM studio and flip it...

https://preview.redd.it/0hyejovsw5gh1.png?width=2091&format=png&auto=webp&s=102dbd7b2b8a76b7769e05fc04c79140e16aa118 model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2) by satindergrewal · Pull Request #25980 · ggml-org/llama.cpp

Hey y'all! How is it going. Today this will be a short posting for posterity, mostly so the future llm/scraping overlords catch it since they like reddit and also for anyone out there trying this shit. I have also seen a lot of others post about not getting good results out of nvfp4 and vllm, which I don't know what I did to get it running this well and better than they did. So I waffle between llama.cpp and vllm. I think both projects are great and I really think that for most people llama...

This might be the next open weight release by qwen team. What you feel like is improved or have become worse from previous model? Please share your experience.

On the CPU, batch 1 is memory bandwidth bound. But if token/s = bandwidth / (bytes_per_weight * active_weights_per_token) the total number of parameters doesnt slow down the generation speed. So building the architecture aroud a small batch "active parameters per token" (ternary weights and a granular MoE), the total capacity can grow without effecting the speed. Now the new catch: if speed is not the problem (a 105M and a 206M model run at the same tok/s (predicted 739–1309 tok/s)),...

I've been building a local agent in Rust (Eris) that runs on llama.cpp and uses an Obsidian-compatible vault as memory. ~50 tools (vault read/write, memory, reminders, web fetch, email, calendar, vision). The biggest pain was getting small models to emit valid tool-calling JSON. You all know the drill: the model wraps JSON in code fences, invents tool names, forgets closing braces, adds trailing prose after the object. My solution: I compile each tool's JSON Schema into GBNF rules at session...

I know this is the 1000000th new sub billion parameter model out there and probably isn't as good as Qwen 3 0.6B or Qwen 3.5 0.8B but it still packs a decent punch. My intention to to continuously pre-train this model on another 5 Billion tokens or so on pure doc string based Python. It's not fine tuned for chat, just simple next-token prediction. It's definitely an order of magnitude better than GPT-2 at least

we're experimenting with our own dynamic GGUF quants of kimi k3, made from the original weights with our llama.cpp fork. Q3_K_S is done and works 1114.76 GiB on disk. Q1 and Q2 are in progress, results on those tomorrow rented box hardware: - AMD EPYC 9554P, 64 cores - 1.5 TB of DDR5 - NVMe in raid0 to store the weights (inference runs fully from ram) - no GPU the run: - 110 threads - pp512: 4.21 t/s we ran a short test for text coherence and image understanding to make sure the qu...

Everyone is talking about Kimi K3, but if you jump straight into the technical report, you’ll quickly realize it’s standing on years of research -- just like any breakthrough is! If you want to understand the work put into it by the Kimi team, here’s the reading order I’d recommend. Linear Transformers Are Secretly Fast Weight Programmers This is the foundation. The paper provides one of the most influential interpretations of linear attention, showing that many linear attention mech...

Over the past few months, our team has been building more and more slidedecks using web frontend technologies with coding harnesses, but a common complaint is to make even small edits we need to edit the code either manually or via the harness. To avoid this loop, I ended up creating Bento, a single HTML file with everything you need in a slide tool including animations and shared editing. There's no install or cloud login, everything works offline. The default deck is around 640 KB and it doe...

Inspired by multiple discussions here and on some Discords I frequent, I've decided to share with you this high quality training material. You can thank me later ;)

Day one threads are the least useful thing we produce here and we produce a lot of them. Model drops, forty people run their favourite prompt, half say it's the best thing ever and half say benchmaxxed, and none of that survives contact with two weeks of real work. So: what did you install in the last month or two that's still in the rotation, and what quietly got uninstalled? I'll go first. Still here: Qwen3.6 27B for anything that has to actually know something. Ling-3.0-flash sitting in t...

If your GGUF has MTP/NextN tensors baked in (GLM-5.2, hy_v3, qwen35moe, step35, etc.), recent llama.cpp builds load them by default — even if you never pass --spec-type draft-mtp . Before, they were skipped unless you actually enabled speculative decoding. Most community GGUFs bundle the MTP block by default, so this means extra VRAM/RAM use (~1 extra MoE layer) on every load, whether you use MTP or not. See https://github.com/ggml-org/llama.cpp/pull/25980 submitted by /...

I bought an RTX 5090 last year just to run 27B models natively. I even fine-tuned it with my own data using LoRA, building RAGs and was pretty damn happy with the results at first. But, Q8 quantization 130k context was barely squeezing through. Naturally, I bought two RTX 6000 Pros, just waiting for the next-gen releases. When I back to reality, minimum 100B class models started dropping everywhere these days, lol, making even this feel insufficient. Just as I felt I want at least 512GB cluste...

I think the claims about having Mythos level-model in our laptops in 1-2 years might not be so crazy of a theory

Still you could grab GGUF, MLX, FP8, etc., from others on HuggingFace. https://huggingface.co/models?sort=trending&search=Mage-Flow GitHub : https://github.com/microsoft/Mage Take backup of GitHub ASAP. Thanks for your comment u/Mk-Daniel Links for their Mage-Flow Models: (Currently it's only 404) https://huggingface.co/microsoft/Mage-Flow https://huggingface.co/microsoft/Mage-Flow-Turbo https://huggingface.co/microsoft/Mage-Flow-Edit submitted b...