This is a HANDWRITTEN post. I spent way too much time trying to get fast inference on Apple Silicon. This post is for people who want to know what's the latest on running local models on their mac, and why they may not be seeing the performance others in the community claim. TL;DR I've spent the last 2 weeks full-time looking into the state of inference optimization on Apple Silicon, and honestly, the software stack is a mess. There is no framework that has all the inference optimizations...