Ecosystem method families
Softprobe does not implement these methods as Softprobe evaluators. They are how frameworks (Promptfoo, DeepEval, judges, human review tools, …) grade agents. Softprobe’s job is to run those frameworks via pinned runners, capture evidence, and gate workflow outcomes.
Use this hub to choose which framework capability you need — then package it as a FrameworkDefinition + RunnerVersion.
Method families
| Family | Examples | Typically owned by |
|---|---|---|
| Deterministic | exact match, contains, regex, JSON Schema | Promptfoo asserts, unit tests |
| Similarity and statistical | edit distance, BLEU/ROUGE, embeddings | Framework metrics / libs |
| Reference-based quality | groundedness, citation, RAG faithfulness | DeepEval / RAG frameworks |
| LLM judge | rubric, G-Eval, factuality, style | Framework LLM-as-judge |
| Comparative judge | pairwise, listwise, tournament | Framework comparative flows |
| Trajectory and tools | tool-used, args, ordering, efficiency | Trajectory metrics + OTEL |
| Environment outcome | tests pass, DB/API/UI state | Env verify in-framework + Softprobe EnvironmentVersion |
| Multi-turn and multi-agent | dialogue quality, handoffs | Multi-turn framework suites |
| Human annotation | rubric, preference, adjudication | Human workflow tools / framework hooks |
| Production and online | sampling, continuous rules, backfill | Softprobe online policy + framework runner |
| Robustness and security | perturbation, red-team | Security suites in-framework |
| Stochastic and repeated | pass@k, variance | Framework trials (see Trials) |
| Meta-evaluation | judge calibration, leakage | Separate framework suites over prior exports |
Softprobe role for every family
text
FrameworkDefinition + RunnerVersion + SubjectVersion + EnvironmentVersion
→ FrameworkAttempt → EvidenceArtifact → optional projection → GateDecisionExtension rule
Add or pin a framework runner that already owns the method. Do not add Softprobe Measurement schemas or Softprobe scorers unless you are building an internal control-plane check (integrity, redaction, capability) — those are not eval authoring.