Skip to content

Tests

Tests are automatically run via GitHub actions. There are two workflows:

  1. test.yml
  2. llm-test.yml

The first workflow is required to pass for all PRs (and it doesn't do any OpenAI calls). The second workflow is required for changes that affect the OpenAI tests (and does actually call LLM). The second workflow requires approval to run. When writing tests that require OpenAI calls, please use pytest.mark.skipif to make them run in only when openai package is installed. If additional dependency for this test is required, install the dependency in the corresponding python version in llm-test.yml.

Make sure all tests pass, this is required for test.yml checks to pass

Running tests locally#

To run tests, install the [test] option:

uv pip install --group test -e ".[openai]"

Then you can run the tests from the test folder using the following command:

just test

Tests whose optional dependencies are not installed are skipped automatically.

Selecting tests#

Tests are grouped by pytest marker, and each recipe carries its own -m filter. Pick the recipe rather than passing -m yourself: pytest's -m is single-valued, so a second one would replace the recipe's filter instead of narrowing it.

  • just test — everything except the LLM-service markers. Needs no API key, no network and no external service: the Docker and Redis suites drive mocks, and a test whose optional package is absent skips itself.
  • just test-llm — every LLM service, or one of them with just test-llm openai. These call the real API and need the provider's key in the environment.
  • just test-all -m "<expression>" — bakes no filter of its own, so an arbitrary marker expression works here, e.g. just test-all -m "not redis".

To mark a new group, add @pytest.mark.<name> to the tests and declare <name> under markers in pyproject.toml.

Coverage#

Any code you commit should not decrease coverage. To ensure your code maintains or increases coverage, use the following commands after installing the required test dependencies:

uv pip install --group test -e ".[openai]"

just test-cov --cov-report=html

Pytest generated a code coverage report and created a htmlcov directory containing an index.html file and other related files. Open index.html in any web browser to visualize and navigate through the coverage data interactively. This interactive visualization allows you to identify uncovered lines and review coverage statistics for individual files.