Tech
A rate limit that counts the tokens you asked for, not the ones you got
Spent a while assuming a 413 meant my prompt was too long. It wasn't. The prompt was 20 tokens.
Groq's free tier caps you at 8,000 tokens per minute, and that budget is charged against the max_tokens you declare , not the number the model actually generates. So this fails:
prompt: 20 tokens, max_tokens: 8192
-> 413 "on tokens per minute (TPM): Limit 8000, Requested 8271"
Nothing was generated. The request was rejected on the ceiling I asked for.
The same model with...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to