flexeval.schema.rubric_schema

pydantic model flexeval.schema.rubric_schema.Rubric

Bases: BaseModel

Show JSON schema

{
   "title": "Rubric",
   "type": "object",
   "properties": {
      "prompt": {
         "description": "Prompt for the rubric.",
         "title": "Prompt",
         "type": "string"
      },
      "choice_scores": {
         "additionalProperties": {
            "anyOf": [
               {
                  "type": "integer"
               },
               {
                  "type": "number"
               }
            ]
         },
         "description": "Choices.",
         "title": "Choice Scores",
         "type": "object"
      },
      "name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "Optional name of the rubric.",
         "title": "Name"
      },
      "notes": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "Optional notes.",
         "title": "Notes"
      }
   },
   "required": [
      "prompt"
   ]
}

Fields:

• choice_scores (dict[str, int | float])

• name (str | None)

• notes (str | None)

• prompt (str)

Validators:

• check_non_empty » choice_scores

• is_rubric_prompt_valid » prompt

field choice_scores: dict[str, int | float] [Optional]

Choices.

Validated by:

• check_non_empty

field name: str | None = None

Optional name of the rubric.

field notes: str | None = None

Optional notes.

field prompt: str [Required]

Prompt for the rubric.

Validated by:

• is_rubric_prompt_valid

validator check_non_empty  »  choice_scores

validator is_rubric_prompt_valid  »  prompt

pydantic model flexeval.schema.rubric_schema.RubricsCollection

Bases: BaseModel

Collection of rubrics that can be used as RubricItems.

Show JSON schema

{
   "title": "RubricsCollection",
   "description": "Collection of rubrics that can be used as :class:`~flexeval.schema.eval_schema.RubricItem`\\s.",
   "type": "object",
   "properties": {
      "rubrics": {
         "additionalProperties": {
            "$ref": "#/$defs/Rubric"
         },
         "description": "Mapping of rubric names to Rubrics. The rubric names are used for matching metrics to specific rubrics.",
         "title": "Rubrics",
         "type": "object"
      }
   },
   "$defs": {
      "Rubric": {
         "properties": {
            "prompt": {
               "description": "Prompt for the rubric.",
               "title": "Prompt",
               "type": "string"
            },
            "choice_scores": {
               "additionalProperties": {
                  "anyOf": [
                     {
                        "type": "integer"
                     },
                     {
                        "type": "number"
                     }
                  ]
               },
               "description": "Choices.",
               "title": "Choice Scores",
               "type": "object"
            },
            "name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "Optional name of the rubric.",
               "title": "Name"
            },
            "notes": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "Optional notes.",
               "title": "Notes"
            }
         },
         "required": [
            "prompt"
         ],
         "title": "Rubric",
         "type": "object"
      }
   }
}

Fields:

• rubrics (dict[str, flexeval.schema.rubric_schema.Rubric])

field rubrics: dict[str, Rubric] [Optional]

Mapping of rubric names to Rubrics. The rubric names are used for matching metrics to specific rubrics.