How we test
This is the whole method. If a test cannot be traced back to these rules, do not trust the result.
Four rules
- Same task. The task is fixed and published before testing starts. Every tool in the test gets the identical task, word for word.
- Same inputs. The same files, the same data and the same starting setup for every tool. We list them with the results.
- One rubric. Criteria and weights are published before scoring. They do not change after the results are in.
- Results linked. Every score links to the task, the inputs and the evidence behind it, so you can check the work yourself.
What this method covers
These four rules apply to every test we publish. They do not apply to explainers, because an explainer does not test anything. An explainer explains how something works, in plain language, with its sources listed. It is always labelled Explainer and it never carries a score.
A Short is labelled Short. When a Short reports a result, it links to the full test that produced it.
How a score is made
Each criterion has a written description of what earns a low, middle or high score. The criteria, the weights and those descriptions are published with the test, before any tool is scored.
A tool can score badly on one criterion and still be worth using. We show the per-criterion scores, not only the total, so you can weigh what matters to you.
What you get with every test
- The task, word for word, and the inputs we used.
- The rubric, with criteria, weights and score descriptions.
- The evidence: recordings, files and outputs for each tool.
- The full score sheet, including the weak results.
- The date of the test and the version of each tool tested.
When we get it wrong
We correct the page, say what changed, and keep the date of the correction. A corrected score keeps its history instead of being quietly replaced.
Judge the results against this page
That is the point of publishing the method. Read a test, then come back here and check it.