Include the token usage for every conversation and workspace

[aponcedeleonch]

Closes: #418

This PR does introduces the changes necessary to track the used tokens per request, process them, and to return them in the API.

Specific changes:

• Make sure we process all the stream and record to DB at the very end
• Include the flag "stream_options": {"include_usage": True}, so the providers respond with the tokens
• Added the necessary processing for the API
• Modified the initial API models to display correctly the tokens and its price

Notable:

• Ollama and Llama CPP tokens cannot be obtained with the current code. It's not critical since these are local providers and shouldn't represent any cost in tokens
  • Ollama: If we switch our requests to not use the Python client but directly HTTP requests we would be able to obtain the tokens
  • Llama CPP: I couldn't find a way to get the tokens when streaming the response.
• Copilot is not returning the used tokens. Also not critical since Copilot charges a fixed price and not per token

Sample response

curl -Ss 'http://localhost:8989/api/v1/workspaces/tokens/token-usage' | jq
{
"tokens_by_model": {
"hosted_vllm/unsloth/Qwen2.5-Coder-32B-Instruct": {
"provider_type": "vllm",
"model": "hosted_vllm/unsloth/Qwen2.5-Coder-32B-Instruct",
"token_usage": {
"input_tokens": 30,
"output_tokens": 10,
"input_cost": 0.0,
"output_cost": 0.0
}
},
"gpt-4o-mini": {
"provider_type": "openai",
"model": "gpt-4o-mini",
"token_usage": {
"input_tokens": 8,
"output_tokens": 10,
"input_cost": 0.0000012,
"output_cost": 0.000006
}
},
"claude-3-5-sonnet-latest": {
"provider_type": "anthropic",
"model": "claude-3-5-sonnet-latest",
"token_usage": {
"input_tokens": 471,
"output_tokens": 35,
"input_cost": 0.001413,
"output_cost": 0.000525
}
}
},
"token_usage": {
"input_tokens": 509,
"output_tokens": 55,
"input_cost": 0.0014142,
"output_cost": 0.000531
}
}