CLI#

FlexEval offers a command-line interface.

flexeval --help
# OR
python -m flexeval --help

The help output contains the list of valid commands:

                                                                                
 Usage: python -m flexeval [OPTIONS] COMMAND [ARGS]...                          
                                                                                
 FlexEval offers a number of CLI commands for convenience.                      
                                                                                
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --log-level                 [critical|error|warning  Log level to use.       │
│                             |info|debug]             [default: info]         │
│ --install-completion                                 Install completion for  │
│                                                      the current shell.      │
│ --show-completion                                    Show completion for the │
│                                                      current shell, to copy  │
│                                                      it or customize the     │
│                                                      installation.           │
│ --help                                               Show this message and   │
│                                                      exit.                   │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ run                 Run FlexEval using the given YAML Eval Run               │
│                     configuration.                                           │
│ summarize-metrics   Print a summary of computed metrics.                     │
│ run-eval-by-name    Run an eval by name.                                     │
╰──────────────────────────────────────────────────────────────────────────────╯

See the Basic CLI usage vignette for a working example of CLI usage.