flexeval.function_types#

Inspection utilities that use type hints to determine the appropriate object to pass to a function metric.

See eval_schema.

Functions

get_acceptable_arg_types(input_type)

get_first_parameter_types(metric_function)

get_function_input(metric_function, ...)

Coerce input_object to a type accepted by metric_function at this metric_level.

get_valid_levels_for_callable(metric_function)

Given a callable, determine the valid metric_level values based on the type annotation of the first parameter.

is_callable_valid_for_metric_level(...)

join_all_contents_to_string(content)

content is a list of dictionaries whose keys include 'content'.
flexeval.function_types.get_acceptable_arg_types(input_type: type) → set[type][source]#
flexeval.function_types.get_first_parameter_types(metric_function: Callable) → set[type][source]#
flexeval.function_types.get_function_input(metric_function: Callable, metric_level: Literal['Message', 'Turn', 'Thread', 'ToolCall'], input_object: Turn | Message | Thread | ToolCall, context_only: bool) → Turn | Message | Thread | ToolCall | str | dict | list[source]#

Coerce input_object to a type accepted by metric_function at this metric_level.

Parameters:

  • metric_function (Callable) – Function to invoke with the returned input.

  • metric_level (eval_schema.MetricLevel) – The metric level at which metric_function is being invoked.

  • input_object (AnyFunctionObjectInput) – The input_object to be coerced, or passed as-is if accepted by metric_function.

  • context_only (bool) – Determines how strings and lists are converted. See schema documentation.

Raises:

  • ValueError – If the function accepts at least one declared type, but

  • it's a type we don't support at all e.g. set or –

  • it's a type we don't support at this metric_level. –

Returns:

The coerced input for metric_function.

Return type:

AnyFunctionObjectInput | str | dict | list

flexeval.function_types.get_valid_levels_for_callable(metric_function: Callable) → set[str][source]#

Given a callable, determine the valid metric_level values based on the type annotation of the first parameter.

Parameters:

metric_function (Callable) – A callable, probably one available via EvalRun

Returns:

Valid values for MetricItem.metric_level

Return type:

set[str]

flexeval.function_types.is_callable_valid_for_metric_level(metric_function: Callable, metric_level: Literal['Message', 'Turn', 'Thread', 'ToolCall']) → bool[source]#
flexeval.function_types.join_all_contents_to_string(content: list[dict] | Any) → str[source]#

content is a list of dictionaries whose keys include ‘content’. Returns a string with all the ‘content’ entries concatenated together, separated by newline.