retrieve_utils
autogen.retrieve_utils.split_files_to_chunks
split_files_to_chunks
Split a list of files into chunks of max_tokens.
Parameters:Name | Description |
---|---|
files | Type: list |
max_tokens | Type: int Default: 4000 |
chunk_mode | Type: str Default: ‘multi_lines’ |
must_break_at_empty_line | Type: bool Default: True |
custom_text_split_function | Type: Callable Default: None |