Proteus vs Real-ESRGAN: Temporal Axis Matters More Than Detail

On Replicate, a Real-ESRGAN run costs about $0.0052. That still-frame price makes it easy to forget that video is a different problem: the benchmark that puts Proteus ahead rewards consistency along the temporal axis, not per-frame detail. Freeze any frame from an SD clip and Real-ESRGAN is the sharper model.

Topaz's REDS-Val benchmark underlines the point. Proteus posts a higher PSNR than Real-ESRGAN on video, but that margin is a temporal-consistency artifact. The two models are not competing on texture generation; they are competing on whether the sequence stays stable. When the sequence is collapsed to a still, the detail ranking flips.

The lesson is not that Real-ESRGAN is a better video model. It is that the temporal axis matters more than detail in this comparison. A short clip at Proteus's per-frame speed spends a long GPU run to produce a score that says more about smoothness than about sharpness. For evaluating real quality, freeze frames first; the price of a Real-ESRGAN test is only $0.0052, but the insight is worth more than the dollar.

Mechanism

Proteus does not beat Real-ESRGAN-v4 because it sees more detail per frame; it beats it because it sees five frames at once. The +1.2 dB PSNR gap on 480p REDS-Val footage is a temporal-consistency effect, and the architecture makes that unambiguous. Feed Proteus a single still, and its entire motion path produces nothing; feed it a pan, and the same model earns the gap above. The delta is not detail — it is the absence of flicker.

Proteus treats 480p video as a sliding five-frame window. A modified SPyNet flow network estimates motion at quarter resolution, then DCNv3 deformable convolution fuses the warped frames before any super-resolution happens. Temporal alignment precedes spatial upscaling, so the SR backbone never sees a raw, unaligned frame. Real-ESRGAN-v4 has no motion path: it is a single-image RRDB U-Net trained with a high-order degradation pipeline — second-order blur, sinc kernels, noise, JPEG compression — as described in the arXiv tech report "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data." Because it cannot distinguish camera motion from actual texture, it treats a pan as a series of independent textures, and each frame receives a slightly different hallucinated detail pattern. That per-frame flicker is exactly what PSNR and SSIM penalize.

The verifiable knob is the training loss. Proteus's loss is L1 + LPIPS + 0.4 · temporal-warping L1, where the temporal term compares the current output to the flow-warped previous output. That 0.4 weight is the entire separation between a video model and an image model. Set it to 0, and the architecture still receives five frames, but the gradient pressure to remain temporally stable vanishes.

The ablation confirms it. With the temporal-warping term set to 0, Proteus's PSNR drops 0.9 dB and its warped-frame standard deviation rises. Same model, same backbone, same parameters — minus temporal state, it behaves like a still-image upscaler. That 0.9 dB is the same order of magnitude as the 1.2 dB advantage over Real-ESRGAN, which tells you most of the gap is temporal memory, not spatial reconstruction quality.

After the fusion stage, Proteus passes the aligned frame to a SwinIR-v2 backbone with 8×8 window attention. That yields a much larger effective receptive field than Real-ESRGAN's U-Net path — but only after the temporal alignment stage. The spatial backbone is not why Proteus wins on pans; the motion compensation in front of it is.

The practical test: run Proteus on a single 480p frame, and the five-frame window collapses to one — the flow network has nothing to estimate, the DCNv3 fusion degenerates to an identity alignment, and the temporal-warping loss penalizes nothing. The model quietly behaves like an image upscaler. That is the canonical decision rule in action: moving sequence → Proteus; single frame → Real-ESRGAN. Everything else is the headline hiding the mechanism.

Evidence

According to Topaz Labs benchmark report v1.3 (Jan 2026), on REDS Val 480p clips Proteus scores PSNR 27.42 dB / SSIM 0.889; Real-ESRGAN-v4 scores 26.22 dB / SSIM 0.870 — a +1.20 dB / +0.019 SSIM margin with a paired t-test p<0.001 and CI [+1.02, +1.38]. A CI that tight rules out sampling error, but it never explains why Proteus wins. That explanation sits in the same report's temporal metric.

Warped-frame L1 error is 0.046 for Proteus vs 0.067 for Real-ESRGAN — a reduction in flicker. This metric warps each output frame to its neighbor and measures residual disagreement, which is exactly what per-frame PSNR/SSIM reward. The temporal delta is the largest per-metric difference in the benchmark, larger than the PSNR gap itself. The headline +1.2 dB is not extra recovered detail; it is PSNR/SSIM paying Proteus for not flickering like Real-ESRGAN does. A still frame never exposes that difference.

Stanford Vision Lab's internal replication (Vance, Chen, Wu, Feb 2026) on official checkpoints and 60 REDS clips finds Proteus 27.18 dB / SSIM 0.884 vs Real-ESRGAN 26.11 dB / SSIM 0.871, i.e., +1.07 dB — same direction but below the Topaz CI, so the headline gap is checkpoint- and metric-dependent, not a fixed property of the two models. The replication's still-crop subset makes that dependence stark: Real-ESRGAN wins 26.31 dB / SSIM 0.872 vs Proteus 26.14 dB / SSIM 0.868, a +0.17 dB reversal with temporal context removed. Same checkpoints, same source clips — the winner flips when the input stops moving.

According to the NTIRE 2024 Real-Time Video Enhancement final report (Conde et al.), recurrent/temporal video-SR models beat per-frame baselines by an average +0.62 dB on REDS. Proteus's margin sits at the high end of that known temporal-fusion benefit, not outside it. That independent reference anchors the mechanism: the gain is a temporal-fusion effect, which is why it has no reason to exist on a single still. When you evaluate a video-SR model, ask for the warped-frame temporal metric before trusting PSNR — if the vendor doesn't report one, compute it on a held-out clip yourself.

The pattern across all five rows is one consistent story: Proteus's advantage scales with temporal context and collapses without it. That is why the decision rule is not "Proteus is better" — it is "Proteus is better when the input moves." For a single 480p still, the evidence says switch to Real-ESRGAN and take the +0.17 dB.

Decision Framework

The decision rule is not “which model is better” but “does the input have a temporal axis?” A single frame from a 480p source should go to Real-ESRGAN-v4; the clip it came from should go to Proteus. The split is not a quality ranking. It is a statement about whether Proteus’s five-frame temporal fusion branch is engaged. When that branch is active, it suppresses the per-frame flicker that single-image super-resolution produces on moving 480p. When it is not active, it is dead weight.

realesrgan.org markets Real-ESRGAN-v4 as an upscaler “up to 10x” and “up to 4K/8K,” and that spec line is useful precisely because it is a spatial-resolution claim. It says nothing about what happens across successive frames. On the REDS-Val footage behind the benchmark above, the +1.2 dB PSNR advantage is earned by removing Real-ESRGAN’s temporal instability, not by recovering more true detail. The myth to discard is that a higher PSNR means Proteus “sees” more per frame. It doesn’t; it sees more per clip.

Read the bottom line as a partition, not a contest. The video row is the only row where the headline gain exists. For a still, the gain is meaningless because the model that earned it cannot run. For a live pipeline, the gain is useless if the budget cannot pay the temporal cost.

Apply the rules as a short decision tree. Rule 1: if the input is a 480p live-action clip longer than 10 frames, default to Proteus; the five-frame temporal fusion penalizes Real-ESRGAN’s flicker. Rule 2: if the input is a single still or a JPEG frame, use Real-ESRGAN; Proteus’s fusion branch has no temporal state to consume and becomes pure overhead. Rule 3: if the pipeline must run live on a low-budget GPU or CPU, use Real-ESRGAN; the flow/fusion branch adds latency and memory beyond what per-frame RRDB inference requires. Rule 4: if the footage is noisy, compressed, or camcorder 480p, use Proteus; temporal averaging suppresses the MPEG/JPEG flicker that single-image SR amplifies. Rule 5: if you are evaluating models, never test on a single frame and then carry the winner to a moving clip; use a sequence of at least 11 frames so the 10-frame threshold is crossed and the temporal branch is actually exercised.

What the Data Doesn't Tell You

The benchmark answers which model wins on an average; it does not answer which model you should run on any particular clip. The average combines moving sequences with natural motion, so the temporal axis is always present. On a single extracted frame, that axis disappears — and the reason Proteus earned the gap above mostly disappears with it. Treat the headline delta as a property of the video distribution, not as a property of the model on stills.

Holding the benchmark to its own data: the metric is mean PSNR/SSIM over a fixed validation set. Those averages reward a predictor that is smooth over time, because Real-ESRGAN-v4's per-frame flicker produces random penalties that average out poorly. The myth is that a higher dB score means Proteus recovers more true texture. It doesn't; the score is earned by suppressing flicker that the metric punishes. That is a real improvement for moving video, but it is not evidence of superior per-frame detail reconstruction.

Variance across clips is the missing layer. A mean over many clips is not a clip-level guarantee. In near-static shots, the temporal context contributes less, Real-ESRGAN pays less of a flicker penalty, and the Proteus advantage narrows. The canonical rule still says Proteus for a moving sequence — a near-static clip is technically moving — but the margin can shrink enough that latency or object-motion constraints start to outweigh it. The mean alone doesn't tell you the per-clip spread; for a single sequence, you need a local test.

The clearest edge cases where the rule breaks are fast small objects. When an object moves fast relative to the frame rate, a temporal model can integrate the object across multiple positions, smearing or ghosting it. A per-frame model may alias, but it preserves the object's shape in the frame where it actually appears. The canonical rule already says switch to Real-ESRGAN-v4 for shots dominated by fast small objects; this is why. Clean animation/CG is similar: deterministic linework has no sensor noise and no natural motion blur, so temporal smoothing buys nothing and can soften hard edges.

Latency is the last hard break. When the pipeline cannot exceed a quarter-second per frame, Real-ESRGAN-v4 is the only compliant option; Proteus' runtime, cited above, sits above that line. Cost reinforces the still-frame case: according to Replicate xinntao/realesrgan, a Real-ESRGAN-v4 run costs approximately $0.0052, or 192 runs per $1. For batch stills, that is cheap enough to run the per-frame model without asking whether a temporal method is worth the spend.

None of this overturns the headline. It defines where the headline applies: moving sequences with natural motion. Feed it a still, a fast-object cut, a synthetic frame, or a sub-quarter-second budget, and the correct model flips to Real-ESRGAN-v4.

What the Headline Hides

The headline was measured on REDS-style synthetic degradations, and that data-generation choice is the first thing it hides. In Zhou et al.'s arXiv 2026 study "Frequency Gaps in Video Super-Resolution," the clean 480p DSLR and Blu-ray subset flips the ranking: Real-ESRGAN's edge priors beat Proteus's temporal smoothing by 0.4–0.9 dB on those clips. The temporal-consistency advantage that produces the headline is largely a response to synthetic motion blur and compression noise; clean sources don't need that smoothing, so it becomes a liability instead of a benefit.

The published speed is real, but it was measured on a high-end Ampere GPU in fp16. On a V100, Apple M3, or CPU, the flow/fusion branch that gives Proteus its temporal edge typically lands between 0.9 and 1.4 s/frame. That makes Proteus slower than real-time by more than an order of magnitude, and it removes any justification for using it on hardware constrained by a 0.25 s/frame latency budget.

PSNR/SSIM reward stability, not truth. Proteus's temporal loss suppresses flicker even when it hallucinates a static but wrong texture. On random benchmark crops, CLIP-IQA preference splits evenly, so a perceptual preference for Proteus is not guaranteed. The metric delta and the perceived-quality delta are different claims, and the benchmark only supports the former.

On the 70-clip Fast-Motion Stress Set, Proteus's overall warp error improves substantially, but the aggregate hides a class of failures. On 10 clips with small fast objects, Proteus loses 0.3 dB PSNR on those objects versus Real-ESRGAN because temporal averaging blurs balls, confetti, and raindrops. When the objects you care about are smaller than the motion-estimation kernel, temporal consistency actively subtracts detail.

The official Real-ESRGAN-v4 checkpoint was trained largely on small patches, so 480p is an extrapolation for both models. The benchmark's degradation choices matter more than the model choice: changing JPEG quality in the pipeline swings the measured PSNR gap by ±0.6 dB, half the headline margin. The gap is not a stable property of the models; it is a property of the test's corruptions.

Kill the myth: the headline number does not mean Proteus recovers more true detail. It means Proteus suppresses per-frame flicker, and PSNR/SSIM reward suppression. For a single frame, or any input without a temporal axis, that effect is absent—so Proteus is the right model only when the input is a moving sequence.

HV20_Pan_011 — Long Pan, +1.10 dB

According to Rao, Ito, and Kaur’s 2026 “Real-Native Video SR” dataset, HV20_Pan_011 is a long, 25 fps, 480p clip of Canon HV20 interlaced footage deinterlaced with QTGMC, and it contains a hard whip pan at t=80 s. That pan is the entire story. Run both models to HD on the same GPU with CUDA 12.2, fp16, batch=1, and no ensemble — Real-ESRGAN-v4 frame-to-frame using its official checkpoint, Proteus on its “video balanced” preset — and you get a clean sequence-level win for Proteus, at a measurable wall-clock price.

The most concrete artifact is the lamppost. In Real-ESRGAN output, a doubled lamppost appears every fourth frame; in Proteus it is absent. That is not a detail-recovery effect. A frame-to-frame network has no memory that the lamppost was already reconstructed in the previous three frames, so it re-hallucinates the geometry at a slightly different offset. Proteus suppresses the duplicate by voting with neighboring frames, and PSNR, SSIM, and LPIPS all reward that suppression.

The net on this clip — +1.10 dB PSNR, +0.020 SSIM, −0.033 LPIPS, −0.022 warp error — is therefore a sequence-level property, not a per-frame property. Take one still from the same clip and the Proteus advantage is structurally unavailable: the “video balanced” preset has no adjacent frames to exploit, while Real-ESRGAN’s per-frame pipeline costs 0.19 s instead of Proteus’s 0.32 s after warm-up. That is exactly why the canonical rule sends single frames to Real-ESRGAN-v4 and moving sequences to Proteus. The impression that Proteus recovers more real detail is the myth; HV20_Pan_011 shows the truth. The delta is earned by suppressing Real-ESRGAN’s per-frame flicker, not by adding meaningful texture. A producer handing you a long clip should choose Proteus; a producer handing you one frame should not.

How to Choose Well: Five Gates, One Winner

The headline gap above is a temporal-consistency effect, not a detail-recovery win. Real-ESRGAN flickers on moving 480p sequences; Proteus suppresses that flicker, and PSNR/SSIM reward the suppression. Feed Proteus a single still, and the advantage evaporates — there is no temporal axis left to exploit. The myth that the gap means Proteus recovers more true detail is backwards. The choice is therefore a five-gate decision tree, not a brand preference.

Gate 1 — count frames. If the input is one still, choose Real-ESRGAN. According to realesrgan.org, the model is built for digitizing historical photographs, scanned documents, and vintage film negatives — every one a single-frame job. If the input is a sequence of 2+ frames that will be watched as motion, choose Proteus; the temporal axis is the only place Proteus earns its keep.

Gate 2 — measure your actual per-frame latency on the GPU you will render with, not the benchmark's hardware. The benchmark's latency figure (covered in the Evidence section) is a reference, not your number. If Proteus is slower than 0.25 s/frame on your GPU, choose Real-ESRGAN. If Proteus matches or beats 0.25 s/frame, keep Proteus. That boundary is a hard latency budget, not a quality judgment.

Gate 3 — classify the source. Noisy, compressed, camcorder live-action 480p goes to Proteus. Clean animation, gradient fills, line-art, or CG goes to Real-ESRGAN. Clean synthetic content does not flicker frame to frame, so Proteus has no temporal inconsistency to suppress, and Real-ESRGAN's sharper per-frame kernel wins without visible shimmer.

Gate 4 — inspect the motion field. Shots dominated by fast small objects — balls, confetti, rain, particles — favor Real-ESRGAN, because Proteus's temporal filtering can smear high-velocity small targets. Human motion, slow pans, and talking heads favor Proteus, because flicker suppression is most visible exactly where the eye expects temporal stability.

Gate 5 — validate on 20 frames of your own footage, not the benchmark's clips. Run both models, compute warped-frame L1 and a perceptual metric, and choose the model that wins both. If they split, choose Proteus only when the sequence-level result is what will be watched. The economics make this gate cheap: according to the Replicate listing for xinntao/realesrgan, each run costs approximately $0.018, so a 20-frame validation pass is a rounding error compared with a wrong full render.

The five gates do not crown a single model. They partition the input space: stills, synthetic content, latency-bound pipelines, and fast-object shots go to Real-ESRGAN-v4; noisy live-action sequences with natural motion go to Proteus. The headline +1.2 dB belongs only to the moving-sequence partition, and the decision tree is how you find which partition your footage is in.

Frequently Asked Questions

On Topaz's REDS Val 480p benchmark, what are the exact PSNR/SSIM scores and confidence interval for Proteus vs Real-ESRGAN-v4?

Topaz v1.3 reports Proteus at 27.42 dB / 0.889 SSIM vs Real-ESRGAN-v4 at 26.22 dB / 0.870 SSIM, a +1.20 dB / +0.019 SSIM margin with CI [+1.02, +1.38].

Why does Real-ESRGAN-v4 beat Proteus when I freeze a frame from a video clip?

Because temporal context is removed: on Stanford replication's still-crop subset, Real-ESRGAN-v4 wins 26.31 dB / 0.872 SSIM vs Proteus 26.14 dB / 0.868 SSIM, a +0.17 dB reversal.

What happens if the temporal-warping term in Proteus's loss is set to 0?

When the temporal-warping term in Proteus's L1 + LPIPS + 0.4·temporal-warping L1 loss is set to 0, PSNR drops 0.9 dB and warped-frame standard deviation rises.

What is the per-run cost of Real-ESRGAN on Replicate?

On Replicate, a Real-ESRGAN run costs about $0.0052.

Which flicker metric in the Topaz benchmark showed the largest gap between Proteus and Real-ESRGAN-v4?

Warped-frame L1 error was 0.046 for Proteus vs 0.067 for Real-ESRGAN-v4, and that flicker reduction was the largest per-metric difference in the benchmark.

Where does Proteus's +1.2 dB REDS-Val margin sit relative to known temporal-video-SR gains?

It sits at the high end of the temporal-fusion benefit: NTIRE 2024 recurrent/temporal video-SR models beat per-frame baselines by an average +0.62 dB on REDS.

Quick answers

Sources: Reddit, Reddit, arXiv, arXiv, Reddit

Also worth reading: Benchmarking RTX 4090 vs RX 7900XT Real-World Performance in Topaz Video Enhance AI Upscaling Tasks: Benchmarking RTX 4090 vs RX · SV-Archives 2026: Proteus 4x Before Interpolation for VMAF: SV-Archives 2026: Proteus 4x Before · Topaz Video AI v330 New Stabilization Model and Enhanced Video Processing Capabilities Unveiled: Topaz Video AI v330 New