ai-agents-map-reduce-chunk-siz-valueerror-a-single-document-was-longer-than-the-c-e1d809b5

THEN Ensure that the chunk_size in your text splitter is less than the token_max parameter in load_summarize_chain. Also set the model's max_length or max_new_tokens to accommodate the largest chunk plus prompt tokens. For example, use RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50) and load_summarize_chain(llm, chain_type='map_reduce', token_max=600).

text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
docs = text_splitter.create_documents([long_text])
summary_chain = load_summarize_chain(llm, chain_type='map_reduce', token_max=600)
output = summary_chain.invoke(docs)