Methodology

CreativeBench compares image and video generation models on practical creative tasks. We assess how well outputs follow a brief, preserve reference details, and meet the task’s quality requirements. Image and video benchmarks are evaluated and ranked separately.

The image benchmark retains the first run for every model, including all required edit steps and paired outputs. Models receive the same task instructions and starting references. In sequential workflows, each model uses its own earlier output where the task requires it. We generate outputs directly through the providers’ APIs, matching output settings where supported. Differences in API capabilities and defaults are part of the conditions under which each model is tested.

We preserve the model’s output without corrective edits or added text or sound. For multi-shot videos, we join full clips in order while keeping their original visuals, duration, and audio. Each video is shown with the prompts used to generate its clips.

Evaluation combines AI and human judgment against task-specific criteria. Images, including edit steps and variants, are reviewed individually. Video reviews assess the full task output, including joined sequences. We compare outputs with the prompt and references through visual inspection and, for video, playback. Criteria cover relevant qualities such as accuracy, task completion, visual quality, motion, continuity, and sound. Ratings and notes explain the assessment.

Each criterion is rated from 1 (not met) to 5 (fully met). The score is the sum of ratings divided by the maximum possible score, expressed as a percentage. This measures performance against the criteria, not the percentage of successful generations.

Overall rankings combine criterion ratings across the benchmark. Tasks with more evaluated outputs or criteria contribute more to the total. Only complete reviews qualify for ranking. Cost and usage are reported separately and do not affect quality scores.

We show API-reported or estimated generation costs and usage where available. These figures cover the recorded generations, not the full production workflow. Missing usage data does not mean zero, and token counts are not directly comparable across providers.