retrieve_utils
autogen.retrieve_utils.split_text_to_chunks
split_text_to_chunks
Split a long text into chunks of max_tokens.
Parameters:Name | Description |
---|---|
text | Type: str |
max_tokens | Type: int Default: 4000 |
chunk_mode | Type: str Default: ‘multi_lines’ |
must_break_at_empty_line | Type: bool Default: True |
overlap | Type: int Default: 0 |