A score needs a setup

Results can depend on prompts, tools, computation budgets and scoring rules. HELM argues for evaluating models across multiple scenarios and dimensions, including accuracy, robustness and efficiency, rather than relying on one number.

Build a small test that resembles your work

FOR A DOCUMENT ASSISTANT

Check correct answers, faithful citations, appropriate “I don't know” responses, latency and cost. Include questions whose answers are absent from the documents.

Keep a held-out set for comparison. Run the same cases after changing a model, prompt or retrieval system. Record failures as well as averages: one severe mistake can matter more than many easy successes.

Read the fine print

Training-data contamination, small samples and subjective grading can weaken conclusions. Company-reported results deserve clear attribution. A higher headline score does not establish better performance for your own language, documents or workflow.

KEEP EXPLORINGMake sense of another idea↗