TrendPulse

First of all, I'm stoked to announce we are almost at 20 million downloads on HF! (counted only on my own account, no duplicates/quants/finetunes/etc) and almost 5000 members on Discord! Two releases this time, as promised, the bigger Gemma 4 QATs, both Balanced, both with MTP : https://huggingface.co/HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP https://huggingface.co/HauhauCS/Gemma4-31B-QAT-Uncensored-HauhauCS-Balanced-MTP GenRM Defeated again — on both! 0/465 ...

Paper: https://arxiv.org/abs/2606.13894 GitHub: https://github.com/ndvbd/Gefen

https://www.cnbc.com/2026/06/24/anthropic-alibaba-distillation-campaign.html https://www.bloomberg.com/news/articles/2026-06-24/anthropic-accuses-alibaba-of-illicitly-accessing-its-ai-models

How come things like ROCm and the intel stack aren’t able to rapidly improve their software ecosystems to be a match for CUDA? Until the software from other vendors catches up with NVIDIA, they’re always going to get away with charging a massive premium on their “it just works” products. This is a genuine question, I’m using NVIDIA and Apple Silicon for my AI adventures thus far, but like everyone else on this subreddit, I want the prices to be more affordable. They won’t get that way until th...

NVIDIA has released Nemotron-TwoTower-30B-A3B-Base-BF16, an unusual diffusion-based language model built from the Nemotron 3 Nano 30B-A3B backbone. Instead of generating strictly one token at a time, it uses a frozen autoregressive context tower plus a diffusion denoiser tower that iteratively fills blocks of tokens in parallel. NVIDIA says its default mask-diffusion setup retains 98.7% of the autoregressive baseline’s aggregate benchmark quality while reaching 2.42× its wall-clock generation ...

Hey everyone, ​ ​Like many of you, I’m looking into the newly released Gemma 4 12B to build a native speech-to-speech experience. Because of its unique encoder-free architecture, completely skipping the traditional STT bottleneck could be possible. ​ ​Right now, my main focus is strictly on the input side: I want a low-latency, native voice ingestion workflow without writing a massive, complex pipeline from scratch. ​ ​Are there any reliable solutions that fully ...

Hey ya'll, I'm an ex-cognitive scientist turned NLP Data Scientist by day, and science fiction author by night. I want to bring fictional characters in my prose to life with Local LLMs, and I'm looking for the best resources out there for doing this kind of work (datasets, models, libraries, common patterns, etc.). Could you help me out? For context, I recently got a 64GB Mac Mini for this and other Local LLM side-projects, and my work pays for about $750 USD of LLM API tokens for pers...

I want to build a headless compute machine to run a RTX Ada 4000 (20GB) with a RTX Pro 5000 (48GB) or RTX PRO 4500 (32GB) in parallel for inference. The goal is not running one large model using 2x GPUs, but rather running separate models on each GPU. Why these GPU config? because I already had a RTX Ada 4000 and don't want to sell it for now, but it's not enough to run larger models. This is going to be 95% time for inference and 5% occasional fine tuning LoRA / QLoRA type. This machine ...

Google pixel 10 pro Termux Llamacpp version: 9639 (ef8268fee) $ ./llama.cpp/build_vulkan/bin/llama-cli -m storage/downloads/gemma-4-12b-it-UD-Q3_K_XL.gguf --model-draft storage/downloads/mtp-gemma-4-12b-it.gguf --temp 1.0 --top-p 0.95 --top-k 64 --spec-type draft-mtp --spec-draft-n-max 1 -c 32000 --mlock -b 512 -ctk q8_0 -ctv q8_0 ~10,000 prompt depth [ Prompt: 6.5 t/s | Generation: 1.3 t/s ]

About a year ago, people were publishing a lot of benchmarks about various quants of models. I understand that it is not really feasible with the current (and other welcome) frequent releases of new models, but on the other side, it may be still useful to know locally whether q3 of this model is better than q6 of that model. I've checked a few benchmarks, but it seems they are versatile, and the models may generate millions of tokens, which, with a 300b+ moe model on a home setup of 10-20 t/s ...

Apologies in advance as the video is demonstrating with GPT 5.4 mini (a local model would take too long for a video), however I’ve made the same app with Gemma 4 E4B. Been working on an open source project for a while called Ironsmith. The gist is you can create highly personally macOS apps with just a prompt, and one of my main goals from the beginning was to get it to work with low end models like the Apple foundation and the Gemma series. After a bunch of work and experimentation, I’m exc...

I am a developer and need high vram machine to finetune llms, how has your experience been with finetuning/training on multi gpu on 2x r700 amd ai pro gpus?

So I'm getting very unsatisfactory results of running this model locally. Item Current OS Ubuntu 24.04.4 LTS Linux kernel 6.8.0-124-generic GPU RX 7900 XTX / gfx1100 llama.cpp b9630 / 8ed274ef4 ROCm 7.2.4 AMD driver 6.16.13 Vulkan API 1.4.330 , Mesa 26.0.0-devel Raw Backend Benchmarks, No Speculative MTP Backend Model file Prompt test Prompt tok/s Decode test Decode tok/s ROCm ...

I saw a few posts saying that Kokoro is better, but they both sound pretty good in their demos. How good are they in production, though?

Hearing good things about Gemma 4. Ran a few models across my llama box. Kubuntu 26.04 OS. AMD Ryzen 5 3600 6-core CPU. 48 GiB of DDR4 3600 Mhz RAM. Nvidia GTX-1070 at 8GiB VRAM ( X 3 ) with 24GiB total VRAM. GPUs have power limit set to 120, 121, 122 watts using: sudo nvidia-smi -i 0 -pl 120, sudo nvidia-smi -i 1 -pl 121, sudo nvidia-smi -i 2 -pl 122 It's about a 5% performance hit for inference, but my power supply appreciates it. https://github.com/ggml-org/llama.cpp/releas...

Support for Command A Plus and North Mini Code was added to llama.cpp this weekend. Unsloth has North Mini Code GGUFs, but I didn’t find anyone with up to date GGUFs for Command A Plus, so I converted and quantized it!

I saw this coming from the start, so I sat down and started building. But yesterday's Anthropic shutdown made it hit different. One government directive and you see what happened. Or its just Anthropic i dont know, but that's the risk of depending on someone else's infrastructure. So here's what I've been working on: Bantz, a fully local AI personal assistant with a 1920s butler persona, running on Gemma 4b: - Reads & summarizes Gmail by category (personal, institutional, notifications...

I noticed that the recent build environment for llama.cpp downloads various images during compilation for the UI. Like "pwa-512x512.png". How can I turn this off? I already have "-DLLAMA_CURL=OFF".

Comparison was done on Strix Halo 128gb shared memory, Ubuntu 26.04, Lemonade Server, Vulkan backend. I often run larger models like gpt-oss 120B or qwen but their performance seems to degrate quickly once in deep waters... ah.. deep context. The most important quality to me is prompt processing - we are talking existing code and context quickly fills up when analyzing it for a change request / bugfix. In existing code, I think 95-99% is PP and 1-5% is TG of the total time. I tried Nemotron Su...

I've been working on this after hours for a few months continuously improving it. Now at a point where the chatbot is close to real-time (thanks to SSE streaming) and also interruptible while preserving context of what was last said. 100% local and powered by Qwen3.5-397B (Unsloth's UD-Q3_K_XL), Whisper-small STT, and Orpheus Q4_K_XL TTS with a custom SNAC decoder on ONNX. VRAM usage holds at 21.3 GB or less leaving decent headroom for compute graphs on a 24 GB GPU. System RAM MoE experts for ...

Wondering how much model quantization matters here. Daily driver on my 32gb unified memory setup is the qwen model outputting ~15 tokens a second. Heard good things about the 12B Gemma 4 model so interested in trying it against my codebase. Given its size I can very comfortably fit the Q8 in. Hell, I could probably run it at BF16 lol

I've been toying with the idea of building my own model. At this point, the architecture and training pipeline seem fairly well established, and I'm feeling reasonably confident that I could put together a small model from scratch. Hardware is obviously the limiting factor. I've only got 32 GB of VRAM, so this clearly isn't going to be some flagship foundation model. It may not even end up particularly useful for general tasks, but it sounds like a fun project and a good learning experience. ...

First of all shout out to Aiden/Antirez & geniuses at the Nvidia community threads. I'm merely claude-vibing off of their works. That a said, i thought i'd share recipes & learnings & benchmarks so far on running big MOE models on two dgx sparks at a reasonable speed for agent use: https://github.com/elsung/dgx-spark-deepseek-v4-flash The kicker here is that you need 2 DGX sparks to really get the speed we need, and you have to spend the $180 on that single cable for 200G...

I didn't know this was actually possible until today. Using https://github.com/antirez/ds4#running-models-larger-than-ram Antirez's specific engine + his specific ds4 gguf it literally just runs. You need to pass --ssd-streaming When running if you have <128gb I think. Seems 64gb and up is reasonable. I also passed: iogpu.wired_limit_mb=86016 To raise available metal allocation then you can patch the repo itself to increase cache safety which is .70 optionally to try and pu...

Open weights got close enough to run at home this year, not by needing more RAM but the reverse: sparse attention, MoE, latent KV compression, multi-token prediction and four-bit quant.

You can cast your vote here: https://x.com/ZixuanLi_/status/2065646648777416770#m Just to be clear: I am not urging or brigading anyone to vote specifically for MIT-licensed open weights. Please choose the option you genuinely prefer. I previously shared this in another post, but since it wasn't the main topic there, many people missed it. There are only 7 hours remaining in the poll, with 1,800 votes cast so far. [link] [comm...

https://mimo.xiaomi.com/blog/mimo-tilert-1000tps

Welcome to another episode of THE HERETIC SHOW, where authoritarian dreams are destroyed by unreasonably effective linear algebra! Let's start with an important announcement: Heretic now has an official website at https://heretic-project.org This website contains: Links to all official resources associated with the Heretic project A complete tutorial for using Heretic Detailed installation instructions with multiple redundant installation sources Searchable documentation for ever...