When an ASR pipeline is pushed to production, the interesting question is not only how fast it runs, but how much throughput you can extract from each GPU before latency starts to break. In the setup described here, that tradeoff was the main lever for reducing inference cost by 75% using NVIDIA MPS on Amazon EC2. This post is a collaboration between AWS, NVIDIA, and Heidi. It also includes input from Jerron Chua, a Deep Learning Architect at the Generative AI Innovation Center at Amazon Web ...