ai-agents-token-tracking-openaicallbackhandler-does-not-record-token-usage--595858b4

THEN Implement a custom callback handler extending BaseCallbackHandler that captures token usage in on_llm_start by extracting the model name from serialized kwargs, then use tiktoken and model-specific cost tables to compute and store prompt/completion tokens and cost. Attach this handler to both the LLM and the AgentExecutor during construction.

class MyTokenTrackingHandler(BaseCallbackHandler):
    total_tokens = 0
    prompt_tokens = 0
    completion_tokens = 0
    successful_requests = 0
    total_cost = 0.0
    model_name = ''
    base_model_name = ''
    price_per_1k_tokens = 0.0
    price_per_1k_tokens_completion = 0.0

    def on_llm_start(self, serialized, prompts, **kwargs):
        self.model_name = serialized['kwargs']['model_name']
        self.base_model_name = 'gpt-4' if 'gpt-4' in self.model_name else 'gpt-3.5-turbo'
        # use tiktoken to estimate tokens from prompts

    def __repr__(self):
        return f'Tokens Used: {self.total_tokens}\nPrompt: {self.prompt_tokens}\nCompletion: {self.completion_tokens}\nTotal Cost: ${self.total_cost}'