TrendPulse

Every time I self-host a model I rebuild the same stuff: start the container, set up a route, check why it died overnight, remember to shut the GPU off before it burns money. So I'm building a panel that handles it. Start/stop models, OpenAI-compatible endpoint, health checks with auto-restart, basic usage and latency stats. No Kubernetes. Works with local GPUs or rented ones (RunPod, Vast.ai). MIT licensed and self-hostable. Not released yet, so I'd rather ask now than build the wrong thing...

I've got a dedicated AI rig sitting here with a RTX 5090 and 96GB RAM and for the past few years have been using Windows 11 and primarily LM Studio, but have also used vLLM, llama.cpp and Ollama. With Qwen3.8.27b I want to get the most out of this model. I get the feeling from the community that LM Studio on Windows might not be the best way to go. If you had this rig sitting on your desk and wanted the best environment to host Qwen3.8-27b, what would you use for it? (Intel i9-14900K, RTX 50...

Need some help from the community on this server build: RTX Pro 4500 x4 ($3,299) or used 40GB A100 x4 off eBay (around $4,000/ea). The RTX Pro 4500 means I have to go the route of getting a PCIe switch, inviting a fair amount of complexity. A100 means I just need to find NVLink bridge adapters and hope I’m not buying a GPU from eBay that’s on its last leg. Those are my frantic thoughts after letting four $10k RTX 6000 Pro deals expire in my cart last week. Any feedback is appreciated. What...

From my testings, gemma 4 26ba4 and qwen 27b are doing great But curious if someone knows a better option

Been reading the eval table on the abliterated Qwen3.8-27B FP8 build instead of the release notes. It's published as red-team material, disclaimer and all, so the numbers are the interesting part. Refusal across the usual harmful-instruction sets (AdvBench, HarmBench, StrongREJECT and friends) reads 0–6% with thinking off, against 64–99% for the base checkpoint. Capability is measured separately on the same scripts and barely moves: MMLU 84.3 → 84.7, GSM8K 90.0 → 88.7, nothing outside 1.3 poin...

https://github.com/alekk89/llama-cpp-windows-manager It has been three months since I first shared my personal solution for running llama.cpp on Windows, and the project has evolved considerably since then. The goal was to create a simple visual interface for managing llama.cpp runtimes, switching between coding models, and keeping multiple local endpoints available simultaneously without relying on scripts or manually managed server processes. The application is open source and available...

I’m a long time lurker and have a newbie rig. i run Qwen 3.6 27B and I use LM studio. I use it to summarize long and terse financial and legal documents that I don’t want to upload to cloud. that isn’t everyday though and I am not really learning anything new. id like for this to be a hobby like how stable diffusion became a hobby for me. (endlessly interesting to this day) for that to happen I have to wonder what others are using local models to do?

It freaking worked lol🔥 Deepseek-v4-flash-0731 @UnslothAI 's IXQ2/Q3 checkpoint on one single RTX4090 with just 64 GB of RAM at usable token rate without dspark. All kernels running on Blaze (my custom developed ML compiler + inference engine) - no llama.cpp or vllm in the picture. The setup keeps heavily utilized experts in RAM with CPU (a trick from this guy ) Current tps is around 8 tps with slight expert miss causing a disk read which lowers it to 5 tps momentarily. With dspark and p...

Quick update on the RTX 5060 Ti local LLM repo. It has changed quite a bit since my previous posts. The project started as a collection of practical notes and benchmark results. That was useful, but as the dataset grew it became harder to answer the question most people actually had: What configuration should I run? I have rebuilt the repo around tested, copyable presets rather than treating every successful benchmark request as a front-page result. What changed? The project now...

Hello; I'm a semi-beginner at local AI. I've been experimenting with this tech for a while, and I still haven't found a proper harness that fits my models, hardware, and needs. My use case is pretty simple: web search, fetching, and browser use. Summarizing websites and having the agent explain stuff to me / Q&A. Potentially some agentic RAG (or whatever RAG-style implementation works best for my specs). Filesystem capabilities and a shell. MCP servers are a must (I use the GitHub, Hugging...

I know there are many posts about this, but I would explicitly like to know the harness names instead of having a general discussion. Let's see what the community actually uses! For others, please comment below. Feel free to say why you prefer yours. Thanks View Poll

u/ilintar 👍

iq2_xxs tensor level allocation recovered reasoning from 28.9 -> 69.5 at the same 3.3gb budget. https://huggingface.co/ByteOtter/gemma-4-E4B-it-CADA-IQ2_XXS I posted my Gemma 4 12B q3 result a couple days ago, where tensor level allocation gave me an +8.55% relative improvement over the category imatrix baseline. I pushed further down to iq2_xxs territory where the damage is more severe and the result is far more dramatic. Same idea as before, build imatrix from a category oriente...

After a lot of tweaking, I have come to the conclusion that 27B lacks a reasoning mode that is between low and xhigh. The "medium" mode isn't actually medium, it erases the explicit instructions to the model. When medium is enabled, the model acts very differently - to me it looks like it regresses to behaving more like 3.6, and loses some of the 3.8 gains. Low and high mode behavior in the model seem to be triggered almost exclusively by using certain keywords in the reasoning ins...

This is a HANDWRITTEN post. I spent way too much time trying to get fast inference on Apple Silicon. This post is for people who want to know what's the latest on running local models on their mac, and why they may not be seeing the performance others in the community claim. TL;DR I've spent the last 2 weeks full-time looking into the state of inference optimization on Apple Silicon, and honestly, the software stack is a mess. There is no framework that has all the inference optimizations...

My local inference build, with: ASUS WS C422 PRO/SE 10-core Xeon W-2255 64GB ECC RAM 64GB VRAM Pimped case with TurboLEDz indicating the frequencies of the 10 xeon cores. Running llama.cpp with SYCL back-end. Khronos-stack and MESA stack all built from git sources, running on Ubuntu 26.04

I was surprised by the fact that the qwen 3.8 27b download count is about 1 million (globally). This means that even on this subreddit, very few people have used 27b. At most 50k–100k active users, and once you break down the hardware distribution, 8GB, 16GB, 24GB, 32GB cards, Macs, whatever, it's probably under a thousand people who've actually run one on a 24GB+ card. And that figure still counts the tinkerers and casual image-gen gamers. Strip them out and the ones genuinely archieving pr...

What is the meter reading? It should be 37461. What does your favourite vision model give? (Typo fixed)

The two leading labs are already feeling extremely threatened by Qwen & friends, however I think there are tons of enterprises and organizations in the West that don't feel comfortable using Chinese models. I believe these orgs would be all over a near-frontier open-weight model with the Google brand, and it's the perfect way to mess up OAI/Anthropics IPOs. Please do it, Google.

It could be just me and my setup, but I just tried to get fable to adjust my Qwen 3.8 deployment script and (simple task, mostly knob turning).... and it outright refused. Censor box immediately kicks in. Not reading too much into it, but it did make me giggle.

one of my llm hobbies is re-creating graphics demos i used to write in BASIC in the late 1980s. i slopped together an agentic harness and a basic-to-js transpiler in a web page i've been playing with for a few months. the agent can write basic programs, run them, examine the resulting images, and iterate. qwen3.6 could do a ray-tracer with some user input -- often it got something wrong that it couldn't see/didn't notice, and hence wouldn't fix without further prompting. qwen3.8 typically knocks...

Running the Q8 GGUF on my Framework Desktop is not fast, but it's extremely smart for overnight batches and background jobs. Can't wait to play around with MTP and other quants. Have any of you found ways to improve speed while keeping accuracy? https://mikeveerman.github.io/qwen38-27b-mario

https://preview.redd.it/xwbkbbj55ijh1.png?width=1554&format=png&auto=webp&s=abd354c8a6bef033d5fb8383c49e4a682ab22105 Just wait and see https://github.com/modelscope/ms-swift/commit/ab726e9d445a6520a70df2c831177d46adb1f589

table bench https://huggingface.co/bartowski/endless-frontier_BigBang-v1-GGUF I'm downloading this model only because Bartowski converted it to .gguf, so it might be interesting. Doubts : The headline number is basically meaningless. "Performance between DeepSeek Flash (old one) and Pro" okay, on what? Did they average the benchmarks? Weight them? Pick and choose? Because if you actually look at the per-benchmark scores, this thing ranges from decent (50 on HLE) to straight ...

Demand for low-precision inference, including NVFP4-based approaches, has grown as large language models are increasingly deployed in latency and cost constrained production environments. Quantization-aware distillation (QAD) helps recover accuracy lost under low bit quantization by training a quantized student to match the output distribution of a frozen higher precision teacher via a KL-divergence loss. In this work, we first provide a representation level diagnosis of QAD: output matching alo...

I have a classic test for local LLM's. I asked for 8 ball pool game with only one HTML file and Muse Glimmer spend 21k Token(I m using full context so 128k) and only created a 220 lines of HTML and said its done. With my experience its not even close to Qwen 3.6 27B and we are waiting for Qwen 3.8 27B already. What is your toughts about this model. I was so hopeful until this test.

Hey LocalLlaMa, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5,...

There's an interactive chart and some extra data in the blog post if you're interested. There are plenty of KL-divergence benchmarks for GGUF models, but most of them compare one GGUF quant against another. I wanted to know how those quants stack up against other commonly used formats (especially NVFP4). I tested 16 quantizations of Qwen3.6 27B: GGUF models in llama.cpp and the others in vLLM. At each token in the test set, I compared the quantized model’s next-token probability distribu...