Getting started

Installation

FlexEval is available on PyPI as python-flexeval.

Install using pip:

pip install python-flexeval

Install using poetry:

poetry add python-flexeval

Install using uv:

uv add python-flexeval

Optional extras

Some functionality depends on optional packages that most users do not need.

To load LangGraph checkpoint SQLite databases (the langgraph_sqlite data source), install the langgraph extra:

pip install 'python-flexeval[langgraph]'

To use LiteLLM completions via litellm_completion(), install the litellm extra:

pip install 'python-flexeval[litellm]'

Usage

Create and run an evaluation:

import flexeval
from flexeval.schema import Eval, EvalRun, FileDataSource, Metrics, FunctionItem, Config

data_sources = [FileDataSource(path="vignettes/conversations.jsonl")]
eval = Eval(metrics=Metrics(function=[FunctionItem(name="flesch_reading_ease")]))
config = Config(clear_tables=True)
eval_run = EvalRun(
    data_sources=data_sources,
    database_path="eval_results.db",
    eval=eval,
    config=config,
)
flexeval.run(eval_run)

This example computes Flesch reading ease for every turn in a list of conversations provided in JSONL format. The metric values are stored in an SQLite database called eval_results.db.

The basic approach:

• Create an Eval defining the functions and metrics that should be computed on the inputs.

• Create an EvalRun defining the input and output data sources.

• Invoke run() to execute the evaluation.

For more information about using FlexEval, continue on to the User guide.

For usage examples, consult the Vignettes.