Tech
Shaving a second off a real-time speech-to-LLM pipeline in Electron
Every part of a speech→LLM pipeline is fast enough on its own. Put them in a row and you get three seconds, which is far too slow when a human is waiting for you to say something.
I build a desktop overlay that listens to the other side of a video call, transcribes it, and streams an answer. The budget I care about is time from the speaker finishing a sentence to the first token on screen. Here is where that second and a half went, and what actually moved it.
The naive pipeline
mic/loop...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to