One prompt, six video models, nine measured checks

What we sent

One prompt, sent unchanged to six text to video models on the same morning, from the same account. Every clip is 16:9 and 6 seconds long, on each model's 720p tier (768p on Hailuo 3, whose smallest tier that is). Sound was left at each model's default. Each model ran twice.

A quiet bakery storefront on a cobblestone street in the early morning. Above the door, a dark green wooden sign reads BAKERY in large white capital letters. A red bicycle leans against the wall to the left of the door. Exactly three terracotta potted plants stand in a row to the right of the door. A small round cafe table with an open yellow umbrella stands on the pavement at the right of the frame. A ginger cat walks steadily along the pavement in front of the shop, from the left edge of the frame to the right edge. The camera slowly pushes in toward the door for the whole shot. Soft low sunlight from the left. No people anywhere in the shot. Photorealistic.

The models: Seedance 2.5, Kling v3, Veo 3.1, Wan 3.0, Grok Imagine Video 1.5 and Hailuo 3. There are no people in the scene on purpose, so every check could be measured by a tool rather than judged.

How we scored it

Nine checks, pass or fail, so a run scores out of 9 and a model out of 18. Eight are measured by tools. One, the count of pots, is counted by eye, because the detector sometimes split one pot into two or counted a plant on a table.

The bicycle, umbrella, sign and layout are checked in the opening third of the clip. The prompt asks the camera to push in, which crops the edges of the scene later, so an umbrella that leaves the frame at second four has not failed.

The scores

The easy parts were easy for everyone. All twelve clips spelled BAKERY right while the sign was in frame, painted a red bicycle and a yellow umbrella, put a row of three pots on the ground, and walked a ginger cat from left to right. The misses were a person nobody asked for, a scene laid out the wrong way round, and a camera that moved the wrong way.

Every clip

Press play on any clip. Nothing loads until you do. First runs:

Second runs, same prompt, same settings:

What the misses looked like

What the checks do not catch

A rubric only sees what it measures. These came up while we checked the misses, and they matter more to some jobs than a pass or fail does.

• How slow is slow. The prompt asked for a slow push in. Measured from 10% to 90% of the clip, Veo 3.1's picture grew 1.8 and 2.6 times; the others grew between 1.10 and 1.45 times, apart from Grok's sideways first run. By our judgement, Veo's move is not slow, and in its second run the sign leaves the top of the frame.
• Lettering nobody asked for. Three of twelve clips added text that is not real words: window lettering in Seedance 2.5's first run, which the text reader read differently in every frame, and menu boards in Veo 3.1's and Wan 3.0's second runs. If your frame has to be clean of text, say so in the prompt and check.
• Open means open. Grok's first run shows a yellow umbrella that stays closed for the whole clip. Our detector finds umbrellas open or closed, so this passed the colour check. We only caught it by eye.
• Where the cat goes. In Veo 3.1's second run the cat turns toward the camera and fills the frame. It still ends up to the right, so it passed.

What you get back

Measured with ffprobe and ffmpeg on the files as delivered. Both runs of each model matched in size, frame rate and length.

Two things stand out. Hailuo 3 on its 768p tier returns a clip longer than asked for, at roughly a quarter of the bitrate most of the others use. Grok and Kling returned tracks so quiet they are close to silent; the prompt did not ask for any sound, so that is a fact to know rather than a fault.

How long each took

What it costs

The rates above are read live from the price list, tier by tier, for a 5 second clip. Our clips ran 6 seconds on the 720p tier (768p on Hailuo 3), so each cost a fifth more than its line here. The price is always shown before you confirm a render.

What this test does not tell you

• It is one prompt and one kind of scene: a still street, one animal, one slow camera move. Hands, fast action, dialogue, several people or long clips can rank the models differently.
• It has no people in it, so it says nothing about how any model draws a person.
• Two runs per model is a small sample. A model that passed both could miss on a third.
• It ran each model at its 720p tier. Higher tiers can change detail, and on some models, the result.
• The detector, the colour tests and the text reader can be wrong. We looked at every miss ourselves and publish the raw numbers below.
• It does not score taste. Light, texture, how natural the cat walks and how the sound fits are for your own eyes and ears.

For movement, we ran a separate test: one standing backflip from the same frame on four models, in Make AI people move realistically. For a car that has to keep its shape, the drive-by test is in A car commercial with AI.

Method and raw numbers

Objects and people: Mask R-CNN trained on COCO, confidence 0.6 (0.7 for people), on 24 frames spread over the clip. Text: the macOS Vision text recognizer with language correction off, on 12 frames. Camera: SIFT features matched between 9 frames from 10% to 90% of the clip, with a RANSAC similarity fit, scales multiplied. Colour: for the bicycle, the most saturated fifth of the pixels inside its mask; for the umbrella, the median hue of the top 40% of its box inside its mask.

Two colour tests changed while we worked, and we say so because they changed results. The first bicycle test counted every saturated pixel in the mask, and the warm wall showing between the tubes outvoted the paint. The first umbrella test failed Seedance 2.5's mustard umbrella on the brick wall behind its pole. Both were fixed to measure the object, then every clip was scored again with the final code. The counts by eye matched the detector on 10 of 12 clips; on both Veo 3.1 clips it also counted a plant on a table or merged two pots.