Quality has several dimensions
Accuracy, robustness to rephrasing, instruction following and cost are different characteristics. The HELM methodology evaluates language models across multiple scenarios and metrics. A single overall score inevitably hides some of these differences.
Build a small evaluation set
For a first experiment, choose twenty tasks from a familiar workflow. This is a useful starting point for investigating errors, rather than a sufficient basis for a universal ranking.
- Write down the success criteria before generating any answers.
- Include ordinary cases, ambiguous cases and questions that cannot be answered from the supplied information.
- Use the same tasks and conditions for every variant being compared.
- Where possible, assess responses without knowing which variant produced them.
- Repeat generation to observe variation.
Look at why errors happen
Track invented facts, omissions, formatting failures and unwarranted confidence separately. Two systems that succeed on the same number of tasks may have very different suitability for your workflow.
Do not tune a prompt against every test example. Reserve separate tasks for the final check; otherwise, you may only measure adaptation to a familiar set.
Record model versions, settings, evaluation dates, inputs and responses. This makes it possible to explain the result and repeat the comparison after the system changes.
Reference on multidimensional evaluation: Liang et al., Holistic Evaluation of Language Models (2022). The twenty-task set is a suggested exercise here, not a HELM standard.