flexeval.data_loader#

Dataset loading functions. Maybe should move to io.

Functions

`add_turns`(thread)
`get_turns`(thread)	We're defining a turn as a list of 1 or more consequtive outputs by the same role, where the role is either 'user', or 'assistant/tool'.
`load_jsonl`(dataset, filename[, ...])
`load_langgraph_sqlite`(dataset, filename[, ...])
`verify_checkpoints_table_exists`(cursor)

flexeval.data_loader.get_turns(thread: Thread)[source]#: We’re defining a turn as a list of 1 or more consequtive outputs by the same role, where the role is either ‘user’, or ‘assistant/tool’. In other words, we would parse as follows: TURN 1 - user TURN 2 - assistant TURN 3 - user TURN 4 - assistant TURN 4 - tool TURN 4 - assistant TURN 5 - user

flexeval.data_loader.load_jsonl(dataset: Dataset, filename: str | Path, max_n_conversation_threads: int | None = None, nb_evaluations_per_thread: int | None = 1)[source]#

flexeval.data_loader.load_langgraph_sqlite(dataset: Dataset, filename: str, max_n_conversation_threads: int | None = None, nb_evaluations_per_thread: int | None = 1)[source]#