ai-agents-embedding-character--cohere-embedding-model-via-aws-bedrock-fails-with--5042cf7b

THEN Configure the chunk_size parameter to ensure each text chunk does not exceed 2048 characters. Since Cohere via Bedrock enforces a character-based limit (not token-based), set chunk_size to a value that keeps chunks under 2048 characters. A workable chunk_size is 80 with chunk_overlap=10, but for larger chunks, implement a custom chunking strategy that splits text by character count using the model's maxLength parameter.

def chunk_text(text, max_length=2048):
    """Split text into chunks based on character count."""
    sentences = text.split('. ')
    chunks = []
    current = ''
    for sentence in sentences:
        if len(current) + len(sentence) + 1 > max_length:
            chunks.append(current)
            current = sentence
        else:
            current = current + '. ' + sentence if current else sentence
    if current:
        chunks.append(current)
    return chunks