flexeval.dependency_graph#
Determines how configured metrics depend on each other.
Functions
|
|
|
Input is the metrics dictionary with keys 'function' and 'rubric', each of which maps to a list Output is list of string representations of the nodes in the graph, in topological order |
|
Input will be a single metric dictionary Output will be - string representation of metric using json.dumps - evaluation_name - function_name or rubric_name |
|
metrics_graph_ordered_list will be a list of metrics in order in which they should be run |
- flexeval.dependency_graph.create_metrics_graph(metrics: Metrics) list[Any][source]#
Input is the metrics dictionary with keys ‘function’ and ‘rubric’, each of which maps to a list Output is list of string representations of the nodes in the graph, in topological order
Each entry and dependency will get an ID so they are easy to match later
- flexeval.dependency_graph.get_metric_info(single_metric: dict) tuple[str, str][source]#
Input will be a single metric dictionary Output will be - string representation of metric using json.dumps - evaluation_name - function_name or rubric_name
- flexeval.dependency_graph.get_parent_metrics(all_metrics: dict, child: dict) tuple[list, list][source]#
metrics_graph_ordered_list will be a list of metrics in order in which they should be run
This function takes the eval represented by “child” and finds ALL evals in “all_metrics” that quality as the child’s immediate parent
An eval can qualify as a parent by having a matching name, type, context_only At this point, we won’t have enough information to decide whether the child should be run (since the child might have additional requirements on the output of the parent) but this is enough to tell us that the child should be run AFTER the parent.