ai-agents-embedding-configurat-using-cohere-embedding-model-e-g-cohere-embed-engl-a06a371b

THEN Implement a character-aware chunking strategy. Reduce the chunk_size to a small token count (e.g., 200 tokens) or use a custom text splitter that limits each chunk to fewer than 2048 characters. Alternatively, switch to a different embedding model like amazon.titan-embed-text-v1 which has an 8k token limit. Ensure that the chunk_size parameter accounts for the model's character limit, not just token limit.

from llama_index.core import Settings
Settings.chunk_size = 200  # Safe token count for English text under 2048 characters
# Or use a custom splitter that enforces character length
from llama_index.core.node_parser import SentenceSplitter
splitter = SentenceSplitter(chunk_size=2000)  # in characters? Adjust per docs