Skip to content

Suite

autogen.beta.eval.dataset.suite.Suite #

Suite(tasks, *, name, source)

An immutable collection of :class:Task records.

Use :meth:from_jsonl to load from disk or :meth:from_list to build inline. Suites are iterable and sized, so they can be passed directly to :func:autogen.beta.eval.run_agent.

Direct constructor — most callers should use :meth:from_jsonl or :meth:from_list.

Source code in autogen/beta/eval/dataset/suite.py 
35
36
37
38
39
40
41
42
43
44
45
def __init__(
    self,
    tasks: Sequence[Task],
    *,
    name: str,
    source: str,
) -> None:
    """Direct constructor — most callers should use :meth:`from_jsonl` or :meth:`from_list`."""
    self._tasks: tuple[Task, ...] = tuple(tasks)
    self._name = name
    self._source = source

name property #

name

Human-readable name (filename stem for JSONL, "inline" otherwise).

source property #

source

Where the tasks came from — a path string, or "inline".

tasks property #

tasks

The tasks in this suite, in dataset order.

from_jsonl classmethod #

from_jsonl(path)

Load a Suite from a JSONL file (one task per line).

Each line must be a JSON object containing at least an "inputs" key (a dict). Missing "task_id" values are filled in from dataset order. Blank lines are skipped.

PARAMETER DESCRIPTION
path

Filesystem path to the JSONL file.

TYPE: str | PathLike[str]

RETURNS DESCRIPTION
Suite

A Suite whose name is the filename stem and whose
Suite

source is the full path string.
Source code in autogen/beta/eval/dataset/suite.py 
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
@classmethod
def from_jsonl(cls, path: str | os.PathLike[str]) -> "Suite":
    """Load a Suite from a JSONL file (one task per line).

    Each line must be a JSON object containing at least an
    ``"inputs"`` key (a dict). Missing ``"task_id"`` values are
    filled in from dataset order. Blank lines are skipped.

    Args:
        path: Filesystem path to the JSONL file.

    Returns:
        A Suite whose ``name`` is the filename stem and whose
        ``source`` is the full path string.
    """
    p = Path(path)
    items = _read_jsonl(p)
    tasks = _items_to_tasks(items, source=str(p))
    return cls(tasks, name=p.stem, source=str(p))

from_list classmethod #

from_list(items, *, name='inline')

Build a Suite from a list of dicts (same shape as JSONL lines).

PARAMETER DESCRIPTION
items

One dict per task. Each must contain "inputs" (a dict). Missing "task_id" values are filled from list order.

TYPE: list[dict[str, Any]]

name

Optional name for the suite, surfaces in the run JSON. Defaults to "inline".

TYPE: str DEFAULT: 'inline'

RETURNS DESCRIPTION
Suite

A Suite whose source is "inline".
Source code in autogen/beta/eval/dataset/suite.py 
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
@classmethod
def from_list(
    cls,
    items: list[dict[str, Any]],
    *,
    name: str = "inline",
) -> "Suite":
    """Build a Suite from a list of dicts (same shape as JSONL lines).

    Args:
        items: One dict per task. Each must contain ``"inputs"`` (a
            dict). Missing ``"task_id"`` values are filled from list
            order.
        name: Optional name for the suite, surfaces in the run JSON.
            Defaults to ``"inline"``.

    Returns:
        A Suite whose ``source`` is ``"inline"``.
    """
    tasks = _items_to_tasks(items, source="inline")
    return cls(tasks, name=name, source="inline")