TrendPulse

So I have been testing a bunch of different solutions for local TTS - nothing so far comes close to elevenlabs for dynamic ability, voices, cloning. I’d like to have a phone-compatible setup. So far the best I can find for edge devices is moss-nano and kokoro. Free/cloud so far : edgeTTS Anyone else have luck so far? Getting their Hermes/openclaw/opencode agents to talk to them via telegram voice note or realtime convo? There’s so many options trying to get them to work is non-trivial...

https://preview.redd.it/9t0qvx6k5z5h1.png?width=1400&format=png&auto=webp&s=88dd83cdd6aa484dcf102bf078f7a80bebb4f7a2 Cypher queries for graph traversal (neo4j) Entity extraction from text chunks (web query, graph query, vectors) Agentic tool calling (Skills selection / successful running in Pi) Code writing (Python) Synthesis/summarization of multi-vector-retrieval Gemma/Qwen in FP8. This brought me joy [link] &#...

Hey everyone! Not a native speaker, so please correct my english where I make mistakes, (can only learn from it!). While it's been out only for just a while, I wanted to post about it because it's been such a joy. So, to say upfront: I use Qwen3.6 27B for programming, Gemma4 for basically everything else. So I can't say anything meaningful about programming. Previously I've used Gemma4-31B Q4_K_L (for long 128k Q8_0 context tasks) and Q6_K_L (for short 32k Q8_0 context tasks). For short ...

Update: you were right to suggest checking the hash. My cached GGUF blob was corrupt. HF expected SHA256: 9188a71055550f1e60b875d02b7abb63625ac11b4a6f148d6b22b3b28ba3d335 My old local blob hashed to: 20e9ffda0c1a0fb5b6ed9cc445834e5c3e98a1f9ffe4a64edf319cbd0aa85fba I moved the blob aside, force-redownloaded with hf download --force-download , and rebuilt latest llama.cpp master after the Gemma 4 MTP merge. Result: main 31B QAT GGUF now works. No more repeated <unused49> . Te...

Hello! I'm using openCode and loading a bunch of different MCP servers at startup. This starts becoming a mess, it eats up tokens and pollutes the context window before I even type a single prompt. How are you all handling this locally? Are you using a proxy/hub to route everything through a single endpoint, or is there a clean way to lazy-load specific tools per session? Curious what the current standard is to keep things clean. Thanks! [link] ...

Running into something annoying with llama-server in router mode (`--models-preset`) and I can't tell if I'm missing a flag or if this is just how it works. My rig is 2x 3090, 2x 4060 Ti (one's unplugged at the moment, riser got repurposed) and a 5060 Ti. I run a single llama-server router that spawns a child per model on demand, which is great. I usually have a few going at once: a 27B at Q8 across both 3090s for coding and my assistant, a little Gemma 4B on the 5060 Ti doing memory/fact-ext...

I wanted something that I could easily configure to manage a set of sensible defaults, that supports multiple llama-server binaries, with per-model over-rides, and command line over-rides. The utility is here: https://github.com/stew675/start-llama I know that llama-server has its own model loading configuration available via the API end-point, but I just wanted something that I could start from the command line easily in one step. I don't know if anyone else may find this useful or not,...

I'm disappointed that ChromaDB's local, free "single node" version is still getting second-class, hand-me-down features while the "distributed" version (a SaaS offering, unsurprisingly) gets built in hybrid search, BM25, etc. I tried to give the benefit of the doubt and wait, but half a year later there's not even an announcement or discussion of feature parity on the roadmap. What are some truly open source, on-premises alternatives well suited to semantically searching a...

So far model releases that take advantage of Quantization Aware Training (QAT) have been focused on 4-bit. I’m curious what could be accomplished with a larger MoE model around 120b up to 400b. Obviously the model could not approach 8/16 bit performance, but perhaps this could be a better alternative to training a ternary LLM (1.58 bit) from scratch. At these sizes you could fit the model into consumer computers running 64/128 gb RAM and perhaps it could out perform a model at about half the s...

Overview: It scored 2% (1.79% rounded up) It is 18/20th place scoring above Haiku 4.5 and Minimax M2.7 Full benchmark took 70 hours Average time per task 32m Average output tokens per task: 44k Perspectives: It scored suspiciously similar to 3.6 Plus and it really gets me wondering how the architecture of 3.6 Plus differs from 27B. Qwen 3.6 27B has a bad reputation in the community for being verbose. But surprisingly. The output tokens were on par or less to similar models....

Hey everyone, Even through I check the subreddit daily, some things are a bit hard to grasp for me due to the speed at progress is made (really impressive!). I tried doing research using deepseek v4 but it left me even more puzzled. Recently I saw NVFP4 support being merged into llama.cpp. Since I have dual RTX 5060 Ti's, I would love to make use of it but I didn't fully grasp how. I also saw someone releasing NVFP4 quants of Gemma4 QAT, seen here: https://huggingface.co/melcheikh/gemma-...

I was originally using Parakeet for my speech recognition pipeline but decided to give Nemotron 3.5 a shot. After testing it on some multilingual audio clips, it's been working great so far. What sold me: - Better language support (40+ locales from one model) - Native streaming architecture — no more buffering entire files - Tested on CPU and got about 4.5x realtime speed using onnxruntime-genai as the backend I've containerized it with Docker so you can just clone and run. There are...

I just came across the following post, where a user found some confusing divergence results between Q4 quants of the original and QAT models with a Q8/unquantized reference of the original model. https://www.reddit.com/r/LocalLLaMA/comments/1tyxu55/gemma_4_31b_qat_q4_vs_standard_q4_top1_kld/ From there I understood that after the retraining of Gemma 4 31B QAT, this could be considered as a different model to Gemma 4 31B original. Therefore, it is not useful to test the divergence of Gemma ...

Title: Gemma 4 QAT MTP assistant heads now public on HuggingFace + PARALLEL=2 crash fix + 12B 2-slot bench (Strix Halo / Vulkan) Three things in one update: the converted QAT-matched draft heads are now uploaded for anyone to use, we found and fixed the PARALLEL=2 crash in both the Atomic fork and filed the same bug on the native llama.cpp PR, and here are the first 12B 2-slot numbers after the fix. 1. QAT-matched MTP heads are now on HuggingFace boxwrench/gemma-4-qat-mtp-assistant-...

Hey everyone! Recently, I released a blog on how to setup a cluster out of your Raspberry Pi 4bs and Mac minis for distributed training and inference Now its time to do the same with Jetson Nano Orin Super! Why ? - 1024 CUDA Cores (Ampere) - 8GB unified memory LPDDR5 - 6x ARM Cortex-A78 @ 1728 MHz, 1024-core Ampere GPU @ 1020 MHz This is a part of my current series where I’ll be releasing blogs and guides around learning distributed learning and building your own small compute cluster...

Hi everyone, I'm new here - because I only have a laptop and I only just realized local models are actually good enough now. So I'd like to share my experience, in case it helps others, and also to learn from the more experienced people here. This is the first model that works for me on my ASUS Zenbook Pro 14 (RTX 4060 8GB VRAM, 64GB RAM): fast enough: ~27TPS generation speed at 32k context, or ~18TPS at 256k context smart enough: it can read and write files, use skills, execute CLI com...

I am using llama.cpp version b9549 with this arguments as recommended: llama-server --temp 1.0 --top-p 0.95 --top-k 64 -hf ... Here is what I got on chessboard svg test https://www.reddit.com/r/LocalLLaMA/comments/1t53dhp/quality_comparison_between_qwen_36_27b/ google/gemma-4-26B-A4B-it-qat-q4_0-gguf:IT google/gemma-4-26B-A4B-it-qat-q4_0-gguf:IT unsloth/gemma-4-26B-A4B-it-qat-GGUF:Q4_K_XL unsloth/gemma-4-26B-A4B-it-qat-GGUF:Q4_K_XL For comparison here is the old gemma4 with...

First strix 495 hardware i have seen announced/leaker. Looks like decent hardware upgraded io. No prices yet that I see sadly.

My x99 just died. F

I built a 3D character you can control with language: https://programasweights.com/avatar Traditionally, 3D avatars are controlled through predefined buttons or scripts. Here you just describe what you want in plain English - including sequences and combinations you'd never wire to buttons, like "wave while walking, then jump a couple times." How it works: it's built on programasweights, which we made earlier that compiles neural programs from plain-English descriptions. This a...

I know local models can’t compete fully yet, but I’m curious about where the limits are. My use case is generating simple HTML activities for elearning creation purposes. I know others are creating apps and more advanced software. Where are the limits for where local models can compete?

I have been a C++ dev for 3 years as long as have done PyTorch in my free time (not that good in the latter). Now, I was lucky enough to get a brand new GPU from a colleague. What are some cool side projects I can build to learn tons about ML and inference/infra? Please don't respond saying "anything you like" as there's nothing I prefer at the moment. I am completely new - so sorry if it's an obvious question! [link] ...

What’s your most unusual or underrated non-LLM AI tool you actually use daily (weird, niche, or non-obvious stuff), and what do you swear by that most people don’t talk about?

Full benchmark results and in-depth analysis are available in the articles: KV Cache Quantization Benchmarks for Long Context and KVarN KV Cache: Implementation and Benchmarks . BeeLlama.cpp (my llama.cpp fork) was used as inference engine due to support of additional types: KVarN (as of v0.3.2 Preview ), q6_0, TurboQuant, and TCQ.

Anyone check out the new Gemma4 12B that dropped 3 days ago? Integrated vision and audio recognition, no mmpro needed plus tool use. Q4 quant is like 8gb RAM. Crazy fast and great quality for it's size. No, it's not as good as a 27B or 31B. But it's damn close. Curious what others think.

Hey r/LocalLLaMA , Some of you might remember when I posted about this project back around September last year (it was called local-deepthink then). The core idea was to move past the usual flat multi-agent setups and instead build something that creates depth . It already ran great locally with llama.cpp or via OpenRouter, and you could export the evolved networks for reuse. But the distillation piece was still coming together. Now in this new mode you fire up a fixed 7-layer QNN topolo...

Just kidding. Are there any distills that actually improve a model's quality? I remember the Qwen R1 8B distill improved the model, but since then, I don't remember ever using a distilled model that was better than the base model. Unless Mythos (or GPT-5.6) is some magical model where only a couple hundred samples will make Qwen-3.6, Qwen-3.5, and Gemma-4 models better I don't care about them. What happened to good distills and why do people only use 250 samples now? submitted by &#3...

I've been running LLMs on my old potato i5-8500 with 32GB of RAM and *no GPU* for awhile now, running up to 12B dense models which run slow but perfectly useable. But this Gemma-4-26B-A4B simply flies on this CPU - only machine using Koboldcpp on Linux. That's right, an old used $150 desktop computer is running state of the art LLMs with something like 7 T/s. Yeah, go ahead and scoff. You can brag about your super-rig that costs more than a used car, but I'm bragging about a crappy old desktop...