flexeval.compute_metrics#

Utilities for computing needed metric computations and actually invoking those computations.

Functions

`add_all_metrics_to_objects`(...)	Adds all metric instances in metrics_for_level to each instance of an evaluable object (e.g., Turn, Thread, Message, or ToolCall) in iterable_of_objects.
`compute_metrics`(evalrun, evalsetrun, datasets)
`count_rubric_metrics`(iterable_of_objects)	Returns the total number of rubric type metrics in the metrics_to_evaluate field in each object.

Classes

`MetricComputer`(function_modules[, evalsetrun])
`MetricGraphBuilder`()	Builds `networkx.DiGraph` s of `ObjectMetric` instances that reflect any computational dependencies between them.
`ObjectMetric`(object, metric)

class flexeval.compute_metrics.MetricComputer(function_modules: list, evalsetrun: EvalSetRun | None = None)[source]#

Bases: object

compute_function_metric(function_name: str, metric_kwargs: dict, input_object: Thread | Turn | Message | ToolCall, metric_level: Literal['Message', 'Turn', 'Thread', 'ToolCall'], depends_on: list, id: int)[source]#

compute_metric(object: Thread | Turn | Message | ToolCall, evaluation_name: str, evaluation_type: str, metric_level: str, kwargs: dict, depends_on: list | None = None, id: int | None = None, notes: str | None = None) → list[dict][source]#

compute_metrics(object: Thread | Turn | Message | ToolCall)[source]#

we’ve defined a variable called metrics_to_evaluate it’s a list we need to loop through each entry looks like this {

‘name’: ‘string_length’, ‘type’: ‘function’, ‘kwargs’: {}, ‘depends_on’: []

}

compute_rubric_metric(rubric_name: str, metric_kwargs: dict, object: Thread | Turn | Message, metric_level: str, depends_on: list, id: int)[source]#

find_function(function_name: str)[source]#

classmethod from_evalrun(evalrun: EvalRun, evalsetrun: EvalSetRun | None = None) → MetricComputer[source]#

invoke_function(metric_function: callable, metric_level: Literal['Message', 'Turn', 'Thread', 'ToolCall'], input_object: Turn | Message | Thread | ToolCall, metric_kwargs: dict)[source]#

load_rubrics(evalsetrun: EvalSetRun) → dict[source]#: Load and return rubrics from the given EvalSetRun.

process_thread_dependency_graph(g: DiGraph, raise_on_error: bool = True) → list[dict][source]#

process_thread_dependency_graphs(graph_list: Iterable[DiGraph]) → list[dict][source]#

class flexeval.compute_metrics.MetricGraphBuilder[source]#

Bases: object

Builds networkx.DiGraph s of ObjectMetric instances that reflect any computational dependencies between them.

build_metric_structures(evalsetrun: EvalSetRun)[source]#

build_thread_task_graph(thread: Thread) → DiGraph[source]#

build_thread_task_graphs(dataset: Dataset) → Iterable[DiGraph][source]#

find_object_metric_from_depends_on(current_object: Message | Turn | ToolCall | Thread, current_metric_level: Literal['Message', 'Turn', 'Thread', 'ToolCall'], current_index: int, depends_on: dict) → ObjectMetric | None[source]#

If you’re a Turn metric that depends on a Message metric, then we create a dependency on ALL or ANY Message meeting the criteria. We don’t know how to handle that…

In contrast, if you’re a Message metric that depends on a Turn metric, then we have a dependency on only a single object: that Message’s Turn.

get_index(target_id: int, objects: list[Message | Turn | ToolCall | Thread])[source]#

get_or_create_object_metric(metric_level: Literal['Message', 'Turn', 'Thread', 'ToolCall'], object: Message | Turn | ToolCall | Thread, metric: dict) → ObjectMetric[source]#

class flexeval.compute_metrics.ObjectMetric(object: Message | Turn | ToolCall | Thread, metric: dict)[source]#: Bases: object

flexeval.compute_metrics.add_all_metrics_to_objects(iterable_of_objects, metrics)[source]#

Adds all metric instances in metrics_for_level to each instance of an evaluable object (e.g., Turn, Thread, Message, or ToolCall) in iterable_of_objects. This addition is done by appending to the metrics_to_evaluate field, which all instances in iterable_of_objects should have.

Parameters:

iterable_of_objects – list of objects that have a metrics_to_evaluate field
metrics – list of metric instances to add to each object

flexeval.compute_metrics.compute_metrics(evalrun: EvalRun, evalsetrun: EvalSetRun, datasets: list[Dataset]) → list[dict][source]#

flexeval.compute_metrics.count_rubric_metrics(iterable_of_objects)[source]#

Returns the total number of rubric type metrics in the metrics_to_evaluate field in each object.

Parameters:: iterable_of_objects – list of objects that have a metrics_to_evaluate field