TrendPulse

Hey r/LocalLLaMA , Wanted to share a narrow fine-tune I've been working on and get some technical feedback from people who've done similar domain-specific work if possible. The problem: general chat models can write marketing copy, but they default to the same tells hedging, "In today's fast-paced world…" openers, vague benefit-speak instead of specifics. Claude is good no dobt about it but I wanted to do something of my own too. I fine-tuned Gemma-4-31B-it specifically to cut t...

I can finetune any model on any dataset I want. I can use techniques like speculative decoding and other sota approaches to get the max tps The llm provides like anthropic and openai are not getting access to my data The hardware is reusable for vision text speech, and I can run any blend of models for free as much as I want I can curate any dataset/content that I want without worrying about the costs I like watching Dario go up in flames

Got another 3090 had to print a bracket to angle the radiator and make room for the GPUs 💀 ended up liking the look more than I thought ..qwen 27b go brrrrr

I’m pretty jaded like most of y’all. I don’t really get excited by new models much anymore. Last few weeks have been kinda meh to be honest. Monday, I stumbled upon SenseNova’s Mixture of Transformers models and they seem kinda like a different animal than other typical image gen models. I managed to get a couple of them running and I have to say that this series of models is impressing me when it comes to generating and editing dense infographics. I haven’t seen anything except for Ideogram...

Sooo... I decided screw it. I'm going to rebuild Gemma 4 31b. I really like the model. So the current plan is to rebuild the SWA layers. Currently running all the proper ablation tests to figure out what SWA layer gets removed. Gemma runs 5 SWA at 1024 tokens each. Then a global layer for the "Block" Layer 3 is consistently the weakest and will likely get removed. From there I am going to rescale the attention of SWA across the board. The new SWA will be 1024/2048/4096/8.1k ...

Hello everyone. I had posted in this sub about making a production service about 8 months ago. Here the link of my previous post . The idea was the same. We wanted to make a real production service that we can provide to clients to earn money. AI assistant that works through messenger, and helps users to work with appointments to the doctors of private clinics. This all devolved into a more than half a year of frustrations and mental agony, and we are finally retiring and shutting down th...

Hi! I'm Andi from Hugging Face. This is a fully open-source and free to test/pull/modify demo I'm bringing today. It's a voice demo creating a pipeline of: - Nvidia's parakeet - Gemma 4 31B (served by cerebras!) - My custom inference for Qwen3TTS It sees and searches the web faster than you blink. The whole stack is fully open-source , and is a drop-in replacement for OpenAI's realtime API. You can run it locally, I get similar latencies with a macbook pro M3 36GB and Gemma 4 E4B. ...

For context, this week they struck a deal to buy Nvidia chips and run local models for their enterprise clients. So in this video he is railing against Anthropic and OpenAI saying they are ripping everyone off while stealing their data too. Always a special moment when the enemy comes around and embraces your world view.

I've been just sit on this thread for a while now, both as a reader and occasional poster, so I figured it was finally time to share something I've been working on last weekends. Google hasn't shipped a dense Gemma4 bigger than 31B, so I decided to just build one myself. Heads up though — I'm not a CS or math person, this is all hands-on trial and error on my own hardware. If anything below is theoretically shaky, please tell me, I genuinely want to learn where I'm wrong. What I did: took ...

My local llama.cpp-based LLM just started reporting this this morning: "DuckDuckGo is blocking with a CAPTCHA. Let me try other approaches:" Is anyone else seeing this with DuckDuckGo?

This is more of a thought experiment than a proposal. I couldn't find much discussion of this in publicly available papers, so I'm wondering whether this has already been explored or whether I'm missing a fundamental limitation. One thing that surprised me after using my Legion Go is how good 800p can look on an 8.8 inch display. It made me wonder whether handhelds deserve a different AI upscaling strategy than desktop GPUs. As handheld hardware tries to keep up with future AAA games, intern...

I'm building a local LLM workstation and would appreciate some advice from people already running 2×3090s. Current hardware: ASUS Crosshair VIII Hero (X570) One Gainward Phoenix RTX 3090 Looking for a second used 3090 (not necessarily the same model) Both GPUs will be power-limited to ~250W I'm trying to keep the case budget under 200 euros SEK (including any extra fans), but might stretch if neccesary... So far I've been looking at: Fractal North XL Mesh (looks nice, but w...

Link to full blog post with all method details, results, and links to all relevant code/skills/prompts at the bottom of this post. Apologies for not having more links throughout, it seems this subreddit restricts too many links in posts. Document redaction tasks are complex tasks that require text and image recognition capabilities, long context understanding, and the ability to closely follow rules to redact specific information. Recently, local models have been developed that can do this al...

hey, I’m currently getting enough VRAM to run something in the GLM-5.2 range, but I’m wondering: do we actually have a solid ranking that compares closed-source and open-weight LLMs side by side? I’ve been trying to find a clear “closed vs open” leaderboard, but most benchmarks feel fragmented or don’t really answer the practical question of what’s actually best to run locally versus what’s only competitive through API models. Also, are there any open models that feel as impressive for th...

Felt like there used to be a thriving fine-tuning community a few years back - and then once we started getting models that were smart enough and generalist enough (i.e. post Llama-3-8b era) things kind of dropped off a little. Less need for fine-tunes when prompt-tweaking can get you most of the way if your base is smart enough I suppose? I do miss it - felt like more or less every week I'd open this sub to find some new weird and wonderful thing going on with home brewed models trained on Unsl...

TL;DR: Small models aren't dumb, they're shallow. I designed a cross-domain, blind, visual experiment to see if a large model can compress its "planning discipline" into a reusable scaffold that makes a small model deeper — with zero fine-tuning. Three.js is the testbed because you can't fake structure with verbose text; the render exposes everything. I’ve been spending a lot of time testing smaller models (like 9B parameters), and I’ve noticed something: they aren’t exactly dumb, ...

After spending countless hours testing on 3 "potato" laptops (Intel i3, 8GB RAM, Win11, integrated GPU), that's my conclusion. For reliably extracting data from images to JSON on low-end hardware, nothing else even comes close. Yet, it’s completely missing from major benchmarks like Artificial Analysis or the Open LLM Leaderboard (while the 4B version is listed). In my (non-scientific) testing, Qwen3-VL-2B Q4_K_M GGUF easily outperforms Qwen3-VL-4B and Qwen3.5 2B for this specifi...

Speculative decoding speeds up LLM generation by using a small "drafter" model to predict several tokens ahead of the main model. The main model then verifies these predictions in a single forward pass. If the main model is heavily quantized (low bit-rate), it becomes less "consistent" with the drafter, lowering the acceptance rate. Models used: Trunk: Gemma 4-31B-it (quantized GGUFs) Drafter: Gemma 4-31B-it-assistant (MTP drafter) Acceptance rate across...

A Sana 1.6B text-to-image transformer compressed to ternary (~1.85 bits/weight): 8.6× smaller than FP16, near-FP16 quality. Footprint (measured) Artifact Size vs FP16 What it is FP16 transformer 3.21 GB 1× (100%) reference Clark Air (packed) 374 MB 8.6× (≈12%) packed ternary ( clark-air-sana-1.6b-packed.safetensors ) Clark Air (unpacked) 3.21 GB compatibility this repo's transformer/ , dequantized bf16, drop-in diffusers Me...

TL;DR: The (very messy) code and writeups can be found at https://github.com/jakint0sh/qwen3-engine Read the README for instructions on how to get started. And for those who just want a bulleted list: - Inference engine for Qwen 3 sizes 4B and below - Written from scratch in pure C - No dependencies except libc, libm, and cJSON (and OpenMP if compiled with parallelization) - Loads directly from HF safetensors, does 4-bit affine quant on the fly - Does KV caching - Built-in chat interface -...

Sharing popular(also recent) models for reference: 151-250B : DeepSeek-V4-Flash Step-3.X-Flash Command-a-plus-05-2026 Laguna-M.1 MiniMax-M2.X Qwen3-235B-A22B 100-150B : GLM-4.5-Air Qwen3.5-122B-A10B NVIDIA-Nemotron-3-Super-120B-A12B Mistral-Small-4-119B-2603 Devstral-2-123B-Instruct-2512 Mistral-Medium-3.5-128B Llama-4-Scout-17B-16E-Instruct (Yay! got your attention) <100B : Llama-3.3-70B-Instruct Qwen3-Coder-Next Qwen3-Next-80B-A3B I ...

DeepSpec DeepSpec is a full-stack codebase for training and evaluating draft models for speculative decoding. It contains data preparation utilities, draft model implementations, training code, and evaluation scripts. Released Checkpoints The checkpoints below are the ones used for Table 1 in the paper . Each checkpoint was trained on open-perfectblend data generated by its corresponding target model in non-thinking mode, and is the direct output of the corresponding training ...

I have been running an open evaluation setup where N models answer the same prompt, then blind-grade each other in an N x N matrix with self-judgments excluded. No single privileged judge. So far: 286 evaluations, 198 hand-written questions, 22,254 valid judgments across 55 models from 11 developer families. Code, dataset, and all prompts are MIT licensed. The finding I did not expect: same-family rating bias is statistically significant in all 8 families with enough data (p < 0.05, 7 of 8 ...

Got all the parts before the crazy price increase except for the rtx pro 5k! Was saving up to order rtx pro 6000 in US and i did, but wanted to join nvidia inception program for the discount. It was around $8.5k during that time, less 1k if I succeeded. It took around 3 months for them just to reject my application which I received last night, and even if i had succeeded in joining, the price already increased to $13.5k, which was beyond my budget. Fortunately, I found 1 rtx pro 5000 in my count...