Basic CLI usage#

All of FlexEval’s features can be accessed using YAML configuration files.

Running an eval is as simple as pointing to that configuration file and using the run() command:

flexeval run vignettes/eval_run.yaml

The file vignettes/eval_run.yaml demonstrates a complete configuration using the EvalRun schema.

 1data_sources:
 2  - path: vignettes/conversations.jsonl
 3database_path: eval_results.db
 4eval:
 5  metrics:
 6    function:
 7      - name: has_question_mark
 8      - name: is_role
 9        kwargs:
10          role: assistant
11    rubric:
12      - name: assistant_asks_a_question
13        depends_on:
14          - name: is_role
15            kwargs:
16              role: assistant
17            metric_min_value: 1
18  grader_llm:
19    function_name: echo_completion
20    kwargs:
21      response: Reasoning.\nNO
22config:
23  clear_tables: True
24  logs_path: vignettes/log
25rubric_paths:
26 - vignettes/custom_rubrics.yaml
27function_modules:
28 - vignettes/custom_functions.py

Use the summarize_metrics() command to print a result summary:

python -m flexeval summarize-metrics vignettes/eval_run.yaml

See Metric Analysis for a more in-depth vignette analyzing FlexEval outputs.