content-summarization-limit-when-using-load-summarize-chain-with-map-reduce-an-d272a9db

THEN Ensure that the `token_max` parameter in the chain is set to a value not less than the maximum token count of any single chunk. Also verify that the model's maximum context length (e.g., `max_length` or `max_new_tokens`) can accommodate the largest chunk. Use the tokenizer to compute token counts of chunks and adjust `chunk_size` accordingly. For example, compute `llm.get_num_tokens(chunk)` and set `token_max` to the allowed context length of the model.

tokenizer = AutoTokenizer.from_pretrained(model_id)
model_max_tokens = 1024
# Set chunk_size so that each chunk has <= model_max_tokens tokens
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
docs = text_splitter.create_documents([long_text])
summary_chain = load_summarize_chain(llm=llm, chain_type='map_reduce', token_max=model_max_tokens)
output = summary_chain.invoke(docs)