VEFX-Reward on test set videos: per-sample predictions vs. human scores
Left: source video; right: edited video. Scores are on a 1–4 scale. Cell shading = |model − human|:
< 0.50.5–11–1.5≥ 1.5
A. Agreement: predictions close to human scores
1 Clean local recolor: agreement at the top of the scale
Instruction. Modify the color of the pink lotus-shaped petals at the base of the statue to a vibrant royal blue. Ensure that the dark red outlines defining the petal shapes are preserved and remain sharp throughout the video. The golden texture and color of the tiered base and the upper statue must remain completely unchanged. This color adjustment should be applied consistently across all frames with no flickering or color bleeding at the boundaries.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 4 | 4 | 4 |
| VEFX-Reward-32B | 3.76-0.24 | 3.77-0.23 | 3.77-0.23 |
| VEFX-Reward-4B | 3.72-0.28 | 3.87-0.13 | 3.85-0.15 |
2 Total failure: agreement at the bottom of the scale
Instruction. Change the color of the man's white t-shirt to a deep navy blue throughout the video.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 1 | 1 | 1 |
| VEFX-Reward-32B | 1.17+0.17 | 1.10+0.10 | 1.22+0.22 |
| VEFX-Reward-4B | 1.14+0.14 | 1.07+0.07 | 1.10+0.10 |
3 Good restyle with side effects: the EE deduction is reproduced
Instruction. Transform the entire video into a fluid watercolor painting style. The scene should feature soft, bleeding color transitions and visible brushstroke textures that mimic paint on wet paper. Ensure the man's silhouette and the specific content on the smartphone screen remain distinguishable while adopting this artistic medium. Maintain temporal consistency so the watercolor textures do not flicker or shift erratically as the video progresses, keeping edges soft yet defined.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 4 | 4 | 3 |
| VEFX-Reward-32B | 3.78-0.22 | 3.73-0.27 | 2.85-0.15 |
| VEFX-Reward-4B | 3.82-0.18 | 3.75-0.25 | 3.34+0.34 |
4 Object insertion that follows the requested path
Instruction. Insert a small red tractor at the start of one of the curved paths, following the contour of the rows.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 4 | 4 | 4 |
| VEFX-Reward-32B | 3.65-0.35 | 3.77-0.23 | 3.81-0.19 |
| VEFX-Reward-4B | 3.37-0.63 | 3.67-0.33 | 3.11-0.89 |
5 Partial success: mid-scale agreement
Instruction. Increase the number of cars driving down the road to make the traffic appear heavier.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 2 | 3 | 3 |
| VEFX-Reward-32B | 2.22+0.22 | 2.90-0.10 | 3.00+0.00 |
| VEFX-Reward-4B | 1.40-0.60 | 3.42+0.42 | 2.23-0.77 |
6 Motion done, content wrong: graded IF
Instruction. Animate the pen to start writing the word 'Inspiration' in elegant cursive across the lined paper. The movement of the pen should be fluid and continuous, mimicking a steady hand-writing motion. Ensure that black ink flows realistically from the nib onto the paper as it moves. The shallow depth of field must be strictly preserved, keeping the pen tip in sharp focus while the hand and background remain softly blurred without any flickering or artifacts.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 3 | 3 | 3 |
| VEFX-Reward-32B | 2.62-0.38 | 3.62+0.62 | 2.39-0.61 |
| VEFX-Reward-4B | 2.34-0.66 | 3.53+0.53 | 2.62-0.38 |
B. Large human–model disagreements
1 Camera Angle Editing · Low angleIF under-estimated
Instruction. Transform the video perspective to a dramatic low-angle shot, positioning the camera closer to the ground to look up at the rhinoceros. The rhino should appear more imposing and powerful as it continues to eat from the stone trough. Ensure the background elements, like the green fence and the dense pine trees, remain stable and sharp with no flickering or distortion as the angle changes.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 4 | 4 | 3 |
| VEFX-Reward-32B | 1.11-2.89 | 3.69-0.31 | 2.61-0.39 |
| VEFX-Reward-4B | 1.90-2.10 | 3.67-0.33 | 2.71-0.29 |
Failure mode: camera-angle edits. The viewpoint lowers gradually while the scene content stays the same. Continuous camera-angle changes like this are hard for the judge to perceive from sparsely sampled frames, so it predicts that no edit was made.
2 Visual Effect Editing · Event effectIF under-estimated
Instruction. Add a gentle rain effect to the scene, creating small ripples on the water surface around the lily pads.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 4 | 4 | 4 |
| VEFX-Reward-32B | 1.68-2.32 | 3.47-0.53 | 3.42-0.58 |
| VEFX-Reward-4B | 1.98-2.02 | 3.51-0.49 | 2.92-1.08 |
Failure mode: small regions. The rain is very subtle: only faint ripples on a small patch of water (about 2% of the frame), which are barely visible at the model's input resolution.
3 Quantity Editing · IncreaseIF under-estimated
Instruction. Increase the number of red Christmas stockings hanging on the fireplace mantle by adding two more identical stockings to the right of the existing four. The new stockings must perfectly match the texture, color, and lighting of the originals to ensure they appear naturally integrated into the scene. Throughout the video, these additional stockings should remain stably attached to the mantle with no flickering or edge artifacts. Maintain the integrity of the stone fireplace background behind the newly added items and ensure the overall lighting remains consistent.
| IF | RQ | EE | |
|---|---|---|---|
| Human MOS | 3 | 3 | 4 |
| VEFX-Reward-32B | 1.54-1.46 | 3.46+0.46 | 3.79-0.21 |
| VEFX-Reward-4B | 1.12-1.88 | 3.25+0.25 | 3.78-0.22 |
Failure mode: small regions. The stockings are small in the frame, and the added one sits inside a row of identical stockings, so the change in count is easy to miss at the model's input resolution.