Quickest way to calculate token usage with streaming responses #7198
-
|
Bit of a novice user here -- accessing metadata doesn't seem to work with streaming responses so is the best + easiest alternative to go with telemetry? Or use some external package like Tiktoken? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
I'm assuming you're using Azure OpenAI. There's currently an open task on the Azure OpenAI SDK to add token information to the response payload for streaming since it currently doesn't exist. The best alternative is what you described (using a package like TikToken). It is, however, a bit difficult to get completely accurate answers though because it's difficult to tokenize the function information provided to the model. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @madsbolaris - Any updates on when this will be implemented? |
Beta Was this translation helpful? Give feedback.
I'm assuming you're using Azure OpenAI. There's currently an open task on the Azure OpenAI SDK to add token information to the response payload for streaming since it currently doesn't exist. The best alternative is what you described (using a package like TikToken). It is, however, a bit difficult to get completely accurate answers though because it's difficult to tokenize the function information provided to the model.