flexeval.configuration.completion_functions#

This file contains a list of Python functions that accept conversations as input and produce conversational turns (aka completions) as output.

When writing a new function, the arguments must include, at minimum: * conversation_history - list of dictionaries with keys (“role”,”content”), whose values are strings * kwargs - dictionary of optional values that can probably be ignored Other arguments can be added, but then must also be specified in the “completion_llm” section of the evals.yaml config.

The outputs must conform to the structure described here: https://platform.openai.com/docs/guides/text-generation/chat-completions-api with the following format:

completion = {
“choices”: [
{
“message”:{

“content”: MY_CONTENT_HERE, “role”:”assistant”

}

}

]

}

Functions

echo_completion(conversation_history, **kwargs)

generic_rest_api_completion(...)

gpt_3p5_turbo(conversation_history, ...)

jan_completion(conversation_history, ...)

litellm_completion(conversation_history, model)

Generate a completion for a given conversation history using LiteLLM's completion().

lm_studio_completion(conversation_history, ...)

no_completion_fn(conversation_history, **kwargs)

open_ai_completion(conversation_history, ...)

Generate a completion for a given conversation history using OpenAI's chat completion API.

open_ai_completion_async(...[, n])

Generate a completion for a given conversation history using OpenAI's chat completion API.

placeholder_completion(conversation_history, ...)

This is just for testing -- always returns 'hi'
flexeval.configuration.completion_functions.echo_completion(conversation_history: List[Dict[str, Any]], **kwargs: Any) → Dict[str, Any][source]#
flexeval.configuration.completion_functions.generic_rest_api_completion(conversation_history, api_key_name: str, endpoint: str, **kwargs)[source]#
flexeval.configuration.completion_functions.gpt_3p5_turbo(conversation_history, model_name, key_name, **kwargs)[source]#
flexeval.configuration.completion_functions.jan_completion(conversation_history, model_name, endpoint, **kwargs)[source]#
flexeval.configuration.completion_functions.litellm_completion(conversation_history: list[dict[str, Any]], model: str, n: int = 1, **kwargs) → dict[str, Any][source]#

Generate a completion for a given conversation history using LiteLLM’s completion().

Parameters:

  • conversation_history (List[Dict[str, Any]]) – The conversation history as a list of message dictionaries.

  • model (str) – The name of the model to use for the completion.

  • n (int, optional) – The number of completion choices to generate. Defaults to 1.

  • **kwargs (Any) – Additional keyword arguments to pass to completion(). Allowed values vary depending on the chosen model.

Returns:

The response.

Return type:

Dict[str, Any]

flexeval.configuration.completion_functions.lm_studio_completion(conversation_history, model_name, endpoint, **kwargs)[source]#
flexeval.configuration.completion_functions.no_completion_fn(conversation_history, **kwargs)[source]#
flexeval.configuration.completion_functions.open_ai_completion(conversation_history: List[Dict[str, Any]], model_name: str, api_key_name: str, n: int = 1, **kwargs: Any) → Dict[str, Any][source]#

Generate a completion for a given conversation history using OpenAI’s chat completion API.

Parameters:

  • conversation_history (List[Dict[str, Any]]) – The conversation history as a list of message dictionaries.

  • model_name (str) – The name of the OpenAI model to use for the completion.

  • api_key_name (str) – The environment variable name where the API key is stored.

  • n (int, optional) – The number of completion choices to generate. Defaults to 1.

  • **kwargs (Any) – Additional keyword arguments to pass to the OpenAI API client.

Returns:

The response from the OpenAI API with unset fields excluded.

Return type:

Dict[str, Any]

flexeval.configuration.completion_functions.open_ai_completion_async(conversation_history: List[Dict[str, Any]], model_name: str, api_key_name: str, n: int = 1, **kwargs: Any) → Dict[str, Any][source]#

Generate a completion for a given conversation history using OpenAI’s chat completion API.

Parameters:

  • conversation_history (List[Dict[str, Any]]) – The conversation history as a list of message dictionaries.

  • model_name (str) – The name of the OpenAI model to use for the completion.

  • api_key_name (str) – The environment variable name where the API key is stored.

  • n (int, optional) – The number of completion choices to generate. Defaults to 1.

  • **kwargs (Any) – Additional keyword arguments to pass to the OpenAI API client.

Returns:

The response from the OpenAI API with unset fields excluded.

Return type:

Dict[str, Any]

flexeval.configuration.completion_functions.placeholder_completion(conversation_history, model_name, **kwargs)[source]#

This is just for testing – always returns ‘hi’