AI video testing
Gemini Omni 1.2 Flash: Testing 15-Second Video Generation and Iterative Editing

Can the new model preserve a character, scene, and small visual details across a short video while requiring fewer attempts and costing less than its predecessor? This article defines a test that can answer that question without guessing at results.
There are no verified run logs, billing records, credentials, or official price data attached to this test brief. Therefore this article does not claim that Gemini Omni 1.2 Flash is cheaper, more consistent, or better than the previous version. It provides a controlled method for finding out.
The question to test
The practical question is not whether one generated clip looks impressive. It is whether the model remains useful after a first draft needs correction. A production workflow usually asks for a sequence, then asks for changes: keep the same person, replace the background, alter one prop, fix a hand, or change the final action.
In this article, video generation means creating a clip from a written prompt or reference input. Iterative editing means making a targeted change to an existing result while attempting to preserve everything else. The comparison is useful only if both model versions receive the same inputs, instructions, and stopping rules.
What the experiment should measure
Record four primary outcomes for every test case:
| Measure | Definition | Evidence to save |
|---|---|---|
| Duration | Actual playable duration of the returned file, in seconds. | Original file, media probe output, and requested duration. |
| Detail consistency | Whether predefined character, prop, wardrobe, lighting, and scene details remain recognizable. | Frame samples, rubric scores, and reviewer notes. |
| Regenerations | Number of completed generations needed to pass the acceptance criteria. | Run ledger with timestamps, prompts, and pass/fail decisions. |
| Cost | Provider-reported charge or a clearly labeled estimate based on documented usage. | Invoice, usage export, or calculation with source and date. |
A fifth measure is useful: edit locality. It records how much of the clip changed after an edit that was supposed to affect only one attribute. A successful edit changes the requested attribute and leaves unrelated attributes stable.
Before running anything
Freeze the comparison before generating the first clip. Write down the exact model identifiers, API version, account or project labels without including secrets, region if relevant, output format, requested duration, resolution, frame rate, and any safety or moderation settings. If a parameter is unavailable in one version, mark the comparison as non-equivalent rather than silently changing the other side.
Never place an API key in a prompt, source file, screenshot, terminal recording, or article. Use an environment variable or the provider’s secret manager. The examples below use placeholder names only; they are not verified commands for a particular vendor SDK.
# Example local configuration. Replace values using the provider's
# current documentation and your own approved test account.
export VIDEO_API_KEY="replace-locally-do-not-publish"
export MODEL_NEW="gemini-omni-1.2-flash"
export MODEL_PREVIOUS="previous-model-id"
export TEST_RUN_ID="2026-11-09-character-edit-comparison"
The model names above are labels for the experiment, not evidence that these exact identifiers or parameters are available. Confirm the current API documentation before executing a request.
Build a small, repeatable test set
Use short cases that expose identity and detail drift. Each case should have one primary question and a fixed acceptance rule. Do not begin with a prompt so complex that a failure cannot be attributed to one cause.
Case A: identity across a 15-second shot
Create one fictional subject with at least five observable attributes: hair shape, jacket color, a distinctive pin, shoe color, and a simple object held in one hand. Describe the subject once, then ask for a continuous shot with a small action such as walking from a doorway to a table.
Subject: fictional adult character, short dark curly hair,
mustard jacket, round blue enamel pin on the left lapel,
white sneakers, holding a red notebook in the right hand.
Scene: a quiet workshop with a tall window and a green desk lamp.
Action: the character walks from the doorway to the desk,
places the red notebook on the desk, and looks toward the window.
Camera: one continuous medium-wide shot, stable camera.
Output: approximately 15 seconds.
Preserve the subject's face, hair, clothing, pin, shoes,
notebook, room layout, and lamp throughout the shot.
Score each attribute at the beginning, middle, and end of the clip. A detail is “preserved” only when the reviewer can identify it without relying on memory of the prompt.
Case B: one-variable iterative edit
Take the accepted or best-scoring result from Case A. Request one change: replace the mustard jacket with a navy jacket. Explicitly instruct the model to preserve the face, hair, pin, shoes, notebook, camera movement, room, lighting, and action.
Edit the provided video.
Change only the character's jacket from mustard to navy.
Keep the same character identity, face, hair, blue pin,
white sneakers, red notebook, room layout, lamp, camera,
timing, action, and lighting.
Do not redesign or restyle the scene.
This case measures whether the editing operation is local. Compare the original and edited clips at matched timestamps. Count unintended changes separately from the requested change.
Case C: prop continuity
Repeat the process with a prop that moves through the scene. Ask the character to open the red notebook, place it on the desk, and close it. This exposes object disappearance, color drift, impossible hand placement, and temporal discontinuity.
Case D: correction after failure
Deliberately choose a clip with one visible defect, such as a missing pin or an incorrect notebook color. Ask for that single correction. Record whether the correction fixes the defect, whether the model introduces a second defect, and whether the result remains usable.
Keep the prompt protocol identical
For a fair comparison, use the same prompt text, reference files, requested duration, output settings, and number of allowed attempts. Run the models in an alternating order when possible: new, previous, previous, new. This reduces the chance that a temporary service condition or reviewer expectation affects one model consistently.
If a model rejects a request, do not rewrite the prompt until the rejection itself has been recorded. Then classify the failure: unsupported parameter, content policy response, invalid media input, timeout, transient service error, or genuine generation failure. A prompt repair may be useful in a separate usability test, but it should not be mixed into the controlled comparison.
Run ledger
Create one row per request, including failed requests. A spreadsheet or line-delimited JSON file is sufficient.
{
"run_id": "2026-11-09-character-edit-comparison",
"case": "B",
"model_label": "new",
"model_identifier": "record-exact-provider-value",
"attempt": 1,
"parent_asset": "case-a-new-attempt-1",
"prompt_hash": "record-if-used",
"requested_duration_seconds": 15,
"output_asset": "case-b-new-attempt-1.mp4",
"status": "success",
"started_at": "record-in-UTC",
"completed_at": "record-in-UTC",
"provider_usage": "record-verbatim-or-null",
"provider_cost": "record-verbatim-or-null",
"notes": "single-variable jacket edit"
}
Use a stable filename that contains the case, model label, attempt, and date. Keep the original response metadata with the media file. Do not overwrite a failed or superseded output.
Verify duration and media properties
Do not infer duration from the request or the filename. Measure the returned file. If FFmpeg is installed, ffprobe can provide a machine-readable check:
ffprobe -v error \
-show_entries format=duration:stream=codec_name,width,height,r_frame_rate \
-of json case-a-new-attempt-1.mp4
Save the command output beside the asset. Compare the measured duration with the requested value using a predefined tolerance. For example, a project may define a pass as 14.5–15.5 seconds, but that interval is an example and must be chosen before reviewing the results.
Also verify that the file can be decoded from start to finish. A successful API response is not sufficient evidence that the file is playable.
ffmpeg -v error -i case-a-new-attempt-1.mp4 -f null -
Sample frames consistently
Review the same timestamps for every clip: 0%, 25%, 50%, 75%, and 100% of measured duration. If the video contains a cut, note the cut and add a frame immediately before and after it. Use the same image size and color-management settings when creating review images.
mkdir -p frames/case-a-new-attempt-1
ffmpeg -i case-a-new-attempt-1.mp4 \
-vf "fps=1/3,scale=1280:-2" \
-frames:v 5 \
frames/case-a-new-attempt-1/frame-%02d.jpg
The command above is only a convenient example; it does not guarantee exact percentage-based sampling for every duration. For a strict audit, calculate timestamps from the measured duration and extract those exact positions.
Score detail consistency without pretending it is objective
Detail consistency is a judgment, so make the judgment explicit. Use a fixed rubric for every clip and have the reviewer score from the evidence, not from the prompt alone.
| Score | Meaning |
|---|---|
| 0 | Missing, replaced, or unrecognizable. |
| 1 | Partially present but visibly changed or unstable. |
| 2 | Recognizable and mostly stable, with a minor deviation. |
| 3 | Clearly preserved at all sampled points. |
Score face, hair, clothing, distinctive pin, shoes, notebook, room layout, lamp, action continuity, and lighting separately. Report the mean only alongside the individual scores; an average can hide a critical failure such as a changed face or vanished prop.
For higher confidence, ask a second reviewer to score a blinded sample. “Blinded” means the reviewer does not know which model produced the clip. Report disagreements rather than silently averaging them away.
Count regenerations consistently
Define success before starting. One possible rule is: the clip must meet the duration tolerance, preserve at least the required identity attributes, contain no critical artifact, and satisfy the edit instruction. If the first result passes, regenerations equal one completed generation. If it fails and the second passes, regenerations equal two.
Separate automatic retries from intentional regenerations. A network retry that returns no model output is an infrastructure event; a new generation request that returns a clip is a model attempt. Record both so operational reliability and creative usefulness are not confused.
Measure cost without inventing prices
Cost cannot be verified from a model name alone. Record the provider’s actual usage unit, the quantity charged, the currency, the account or project scope, and the timestamp or billing-period reference. If the provider exposes only an estimate, label it as an estimate.
Use this calculation only after confirming the provider’s current pricing definition:
total_cost =
sum(completed_generation_charge)
+ sum(edit_generation_charge)
+ sum(retry_or_processing_charge)
cost_per_accepted_clip =
total_cost / number_of_accepted_clips
Do not compare a displayed list price for one model with an invoice total for another. Normalize the comparison to the same number of accepted clips and disclose whether taxes, credits, subscriptions, storage, or post-processing are included.
If no billing data is available, report cost as “not verified.” That is more useful than inserting an unconfirmed price into a benchmark.
Suggested result table
Populate this table only from the run ledger and saved evidence. Blank cells are preferable to invented values.
| Case | Model | Measured duration | Detail score | Attempts | Verified cost | Pass? |
|---|---|---|---|---|---|---|
| A: identity | New | To measure | To score | To count | To verify | Pending |
| A: identity | Previous | To measure | To score | To count | To verify | Pending |
| B: jacket edit | New | To measure | To score | To count | To verify | Pending |
| B: jacket edit | Previous | To measure | To score | To count | To verify | Pending |
Failure cases to document
A reproducible report includes failures. Record the exact response class and the asset state for each one.
- Requested duration is ignored or clipped.
- The returned file cannot be decoded or has no video stream.
- The character changes between frames or after an edit.
- A small edit changes the room, camera, lighting, or action.
- Hands, faces, text, logos, or small props become visibly corrupted.
- The request times out after billing or usage is recorded.
- A retry returns a different result without a documented prompt change.
- The provider changes an API parameter or output format during the test window.
Do not delete failed outputs. Hash them, preserve the response metadata, and mark them as failures in the ledger. If a failure exposes personal data or a secret, redact the publication copy while retaining a secure internal record according to the project’s policy.
How to interpret the comparison
A model should not be called “better” because it wins one attractive example. Look for a pattern across cases: higher median detail scores, fewer accepted-clip attempts, fewer unintended edit changes, and a verified cost advantage under the same workload.
Use medians for attempts and cost when a few difficult cases create outliers. Report the full distribution as well. A model that is slightly cheaper on average but fails badly on one critical edit may not be the better production choice.
Treat any conclusion as conditional on the tested settings. Duration, resolution, reference-image support, prompt format, account tier, regional availability, safety filters, and service load can all affect the outcome. A result from four handcrafted prompts is a smoke test, not a general benchmark.
Limitations
This article contains a protocol, example prompts, measurement commands, and reporting templates. It does not contain verified generation outputs, provider billing records, authenticated API calls, or a completed comparison. No claim about the relative quality, speed, availability, or price of Gemini Omni 1.2 Flash can be made from this document alone.
The largest remaining limitation is evaluator agreement. Visual continuity is partly subjective, especially when a change is subtle. A stronger study should preregister its acceptance rules, use blinded reviewers, include repeated runs, publish anonymized metadata, and preserve the exact output files used for scoring.
Final checklist
- Record exact model identifiers and API settings.
- Use the same prompts, inputs, and stopping rules for both versions.
- Save every output, including failures and retries.
- Measure actual duration with a media probe.
- Sample matched frames and score predefined details.
- Count completed generations separately from infrastructure retries.
- Use provider billing evidence for cost, or report cost as unverified.
- Publish limitations alongside any conclusion.
For related terminology, see the AI video glossary. For more practical workflows, continue to the Agent Lab Journal guides.