Methodology

One variable at a time

An experiment changes one thing and holds everything else fixed. Seed, prompt, input image, resolution, frame count and step count stay constant across the runs inside a single experiment, and the record lists all of them. When two things change at once, the result explains nothing, so it does not get published as a comparison.

What gets recorded

Every experiment record contains, at minimum:

• Platform, GPU or SoC, and the amount of VRAM or unified memory.
• Operating system, driver version, runtime version, Python and PyTorch versions.
• Model name including quantization, and any LoRAs loaded.
• The controlled conditions, including the seed.
• One entry per run with its outcome, peak memory, and timing.
• At least one artifact: an output file, a screenshot, or a log.

Records that do not carry those fields fail schema validation and never reach the site. The full schema and the records themselves are in the public repository.

Outcomes

A run ends in one of four states, and all four get recorded:

Success

The run finished and produced usable output.

Out of memory

The run hit the memory ceiling. The exact error text goes in the record, because that string is what people search for.

Error

The run failed for a reason other than memory. The error message goes in the record.

Degraded

The run finished, but the output was bad enough to matter. Degraded is a judgement call, so the record says what was wrong with the result.

How memory and timing are measured

Peak memory is the maximum reported during the run, not the steady state, because the peak is what decides whether a configuration fits. Generation time covers the full run from queue to written output, which includes model loading on a cold start. When a number was taken with the model already resident, the record says so, since the difference on a 12GB card is large.

Token rates for language models are measured at a stated context length, and the context length is part of every row in the database. A token rate quoted without one is not comparable to anything.

Single runs, not averages

Most records here are single runs. Repeating a video generation ten times on consumer hardware costs hours and rarely changes the conclusion, which is usually about whether something fits in memory rather than whether it is 4 percent faster. Where run-to-run variation matters, the record contains multiple runs and the article says so. Nothing here is presented as a statistically averaged benchmark, because that is not what it is.

What these numbers do not tell you

A measurement describes one machine, one driver version, and one runtime build. A different driver can move peak memory by enough to change a success into an out-of-memory error. Thermal behaviour differs between a desktop card and a laptop SoC under sustained load, and a laptop number taken cold does not describe the tenth run of the evening.

Every article carries a section on what would change on other configurations. When that section cannot be written honestly, the measurement is too narrow to publish.

Re-verification

Records carry a verification date and are marked stale after 90 days. Re-verifying means running the experiment again against current versions and recording the result as a new run, keeping the old one. The history of a record is part of the record, and rows are not overwritten to look fresh.