observability-token-usage-tracking-openaicallbackhandler-does-not-record-token-usage--f0fdfd14

THEN Create a custom BaseCallbackHandler that overrides on_llm_start to capture the model name, then compute token counts and cost manually using tiktoken. Pass an instance of this handler to both the LLM and the AgentExecutor.

class TokenTracker(BaseCallbackHandler):
    total_tokens = 0
    prompt_tokens = 0
    completion_tokens = 0
    successful_requests = 0
    total_cost = 0.0
    model_name = ''

    def on_llm_start(self, serialized, prompts, **kwargs):
        self.model_name = serialized['kwargs']['model_name']
    def on_llm_end(self, response, **kwargs):
        # Use tiktoken to count tokens from prompts and responses
        # Then calculate cost based on model pricing
        pass
    def __repr__(self):
        return f'Tokens: {self.total_tokens}, Cost: ${self.total_cost}'