TrendPulse

I'm writing a short series of tutorials on FlashAttention: from theory to efficient CUDA kernels. Part 1 is the theoretical foundation. It walks through a modern algebraic formalism showing that FlashAttention is an associative operation, which lets you treat it as a regular reduction on the GPU and apply all the same scheduling optimizations. Some recent MLSys and CVPR papers lean on this framing, and I find it much more powerful than the original. Overview: Safe softmax, Welford's vari...

So I decided to learn how to fine-tune LLMs. Read a few guides from Unsloth, poked around, then stumbled on Unsloth Studio and wanted to test it out. The dataset I started from a set of relatively unrelated QA pairs — Natural Questions — and stripped the answers. Then I had DeepSeek v4 Pro (thinking disabled) repopulate them: - 1000 train + 200 val = 1200 requests total, cost $0.36 (~$0.0003/req). Honestly impressive on DeepSeek's side. Unsloth Studio It's a huge pain in the butt —...

So I figured I'd update the community given I just shipped a nice little feature set and feel like sharing it finally :) In the past few weeks, I've been test-coding an isometric RPG game/engine in Three.js, as part of my research into how LLMs work at scale in higher quality projects written from scratch (spoiler: they don't, even the SOTA ones). For that, I needed a complete team of virtual creators ;) and working through the Python pipelines for all those models is insanely frustrating (bon...

I noticed while using both, mimo was often better, after benchmarking mimo v2.5 via open code endpoint in diff harness like codex, oh my pi, hermes. i found that mimo is indeed better in coding tasks. and over all, hermes scored 55% with mimo v2.5 via terminal bench v2.0 others did under 50% too with any harness from my list or deepseek v4 flash Not that i dont like deep seek v4 flash its GOAT, i have used it more. but as per benchmark both models are same at most places but when u run real...

Crossposting this here because I thought you guys might appreciate it. When ChatGPT and other open source LLMs first came out, there was a lot of speculation as to how these technologies could change gaming. I recall there being posts and comments about when we could have AI powered NPCs. Nvidia showcased ACE back in 2024, which was an NPC powered by a cloud LLM server. Fast forward to today, and there's a lot of doom and gloom around AI, rightfully so in the case of pretty much every closed ...

Hey guys, Despite browsing this sub, and the myriad of other LLM reddits, I feel no closer to understanding what the best memory system is these days. I doubt this problem is "solved" by any means, but surely it's gotten more a better solution than just say Obsidian right? Currently I'm using the OKF files and it seems to be working ok. But I haven't been using my local model long enough to really see the problems. What's the consensus right now? submitted by ...

okay so you might be following me or not.. but I have been working in AI since last 10+ years and our first product in AI was released in 2014 https://web.archive.org/web/20141027082348/http://xepan.org/ and we have to take that out as it was just not accepted. Now with this new wave of AI I also started picking my pace. And found that training is okay but running a llm is costly and all models are variants of transformers in one or other way. So I tried with some maths first and some theo...

Been using this for real software development for a commercial app. i.e. Not a single file HTML app. I mean a large scale 100k+ loc project that needs proper architecture to work with in a maintainable way. As much as I love Qwen3.6-27b. It just does not understand software architecture, it will happily write spaghetti code, mix concerns, and totally ignore any kind of test automation unless you explicitly ask it to do this. These are the bare minimum requirements for production code that can ...

Warning: I am an accountant and not an ML engineer of any kind, and I'm potentially missing some important points. I wrote all this by hand, but I'll link my gemini chat where I was trying to understand this at the bottom so y'all can decide if I've got AI psychosis or not. I was reading through Anthropic's latest publication on the "J space" and trying to translate it to dumb dumb terms that my 3blue1brown-pilled brain can comprehend, and I think I'm grasping the core concepts, t...

I reiterated on the previous comparison but this time compared different quants of the same model. Same prompt: Write a single HTML file with a full-page canvas and no libraries. Simulate a realistic Döner Style kebab skewer rotating (vertically) in front of a gas powered heating element. Especially Gemma 4 looks more lobotomized the lower you go, but the others also lost "finesse" (no turning, simpler fire and with IQ2 stuff is mostly all over the place, these are the BEST...

Recently bought into the local LLM hype by buying a 32gb vram gpu and holy shit, gemma 4 31b at 5bits blows the standard ChatGPT model out of the fucking water. I just can't unsee the quality difference now that I've experienced it. Does Openai just cut costs by serving their customers this slop since most people just use it as a search engine?

Our new open-source AI model for Ancient Egyptian hieroglyph translation. Available in two versions: Horus Hiero 9B Horus Hiero Mini 4B (optimized for CPUs and mobile devices) Built on Qwen 3.5 , Horus Hiero supports ~150 languages , understands text, images, and video , and is the first model of its kind to combine large-scale multimodal capabilities with dedicated hieroglyph translation. It also delivers strong general reasoning and coding performance: 79% on MMLU-Pr...

https://codepen.io/Captain-Blackbeard/pen/EaZQKWX prompt: "Task: create a beautiful, relaxing flight simulator in a single html page" harness: opencode environment: empty model: hy3 (free) via openrouter

My goal is to improve as a developer, thus I needed to know if local llms can answer technical questions accurately The conclusion is that without rag they don't do too well, but with rag they are very good. Thinking didn't really help, and took so long I only got the scores for e2b and e4b, the rest are still running, it was like only +1% point for thinking. This is what I did: - Downloaded the markdown docs from the github repos for the listed projects (Node, Langchain.js, typescript, t...

https://www.theinformation.com/briefings/exclusive-chinas-minimax-plans-launch-2-7-trillion-parameter-model According to The Information, MiniMax plans to launch a new-generation large language model with 2.7 trillion parameters. Sources revealed that the internal codename for this new model is M3 Pro. It is expected to be released and open-sourced as early as the third quarter of this year, with significant improvements in handling complex reasoning and multi-step tasks. This new model i...

I just released a research project I did last year as open source. It is a fully local speech-to-speech backend for LLM NPCs. So speech-to-text, local LLM, text-to-speech, no cloud needed. The main focus was NPCs talking to each other, not just answering the player, and my study looked at how players experience witnessing those NPC-to-NPC conversations and what it does for immersion. The NPCs can talk to each other, remember what they said, and later use that context when the player talks to...

this was a long time coming, but it's finally here! you can now basically supercharge whichever UI you're already using with the power of openlumara . click that link for more information about openlumara itself. TL;DR: super token efficient framework built from the ground up for local models, reinventing a lot of conventions about harnesses and agents that were made for cloud API's and which tend to make local models work badly. see the link for more info on how it works with the quirks of loc...

I've been a lurker for a while and have been building my own home lab with P40's and MI50's. I've learned so much from the community and I just felt like it's time to give back. Even though I'm still learning I'm sure this information will be valuable to someone out there. I'll be posting MI50's details once I'm done fine tuning my P40 box. Hardware: Asus X99-E-WS ( Modded BIOS to support a large number GPU's ) Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz 128GB DDR4 RAM (mixed batch of Non-E...

If you generate structured output from an LLM and validate against a schema, you know the failure mode: usually fine, occasionally a missing field or an unparseable response. The common fix is a retry, but a plain retry is the same prompt at the same temperature, so you are just re-rolling. What worked much better for me was making the retry self-correcting: when validation fails, put the validation error and the model's own previous output back into the next prompt and ask it to fix that spec...

Just wanted to share that I was looking for an optimal way to run Ornith 35B in FP8 with E4M3 and MTP with vLLM but there was no out-of-the-box model with MTP drafter support. So I grafted this new model! It's 18% faster than without MTP and the drafter acceptance rate is not bad (70% on avg). It should run on any RTX based setup > 80GB VRAM with full context window 256k. Might also do well on Unified Memory Systems like GB10 (for this - use my script and graft the MTP model into a target N...

I want to up my game, with RAG. I tested it ages ago, but haven't found a usecase. I play with coding, projects, and light sysadmin work. # Thoughts RFC library - seems verbose unnecessary industry standards - typically in the model better than my cherry picked documents Codebase - I don't have the largest code base (fits in context) and it changes too often to index? Entire API references - this might work for small scripting languages, but for a bigger language (c#, nodejs, etc etc) ...

I will definitely use vLLM now (unless there is something faster now) but i want to make sure ggufs + llamacpp works along with comfyui and things of that nature too.

We need more of this, 100+ T/s on dense models is the difference between defaulting to Claude/Codex for everything vs having a local private model doing most of the heavy lifting and only reaching for frontier for heavy intelligence work. https://x.com/xenovacom/status/2065656427117437213

I came across this interesting article https://blog.exolabs.net/nvidia-dgx-spark/ while I don't have the DGX spark but it made me curious will this kind of arch speed up my setup for LLMs? Mac can host large models but the prefill speed sucks, so I tested in it on my setup for Kimi 2.7. Short answer: it helps prefill, but it does not meaningfully help decode on this setup. RPC is still mostly a capacity tool unless the network/interconnect and split mode are much better. Setup Host:...

Im on the bus to work and just felt like i dont see enough grattitude for the men, women, children, and people who contribute thier time and effort on open projects. Just last night i saw ive been sleeping while vllm developers are releasing 3 new major releases, and not only that, the issues with OOM caused by preallocations and tuning seem to be gone!!!amazing!! Just fixing that bug has allowed me to double my context window size 120k to 240k with qwen27b on 5090. So this is just a reminde...