split_files_to_chunks

split_files_to_chunks(
    files: list,
    max_tokens: int = 4000,
    chunk_mode: str = 'multi_lines',
    must_break_at_empty_line: bool = True,
    custom_text_split_function: Callable = None
) -> tuple[list[str], list[dict]]

Split a list of files into chunks of max_tokens.

Parameters:
NameDescription
filesType: list
max_tokensType: int

Default: 4000
chunk_modeType: str

Default: ‘multi_lines’
must_break_at_empty_lineType: bool

Default: True
custom_text_split_functionType: Callable

Default: None