Project
Redrob Image
44-16-30 over 90 blind pairs (internal raters)
At n=90 the interval is roughly ±10 points: large gaps ruled out, parity not established. Decision study, not a published claim. Latency ~1.8s end-to-end on one g7e.2xlarge (NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB; model resident, queue excluded). Studio is the client; this page covers the model path.
Summary
Open-weight Z-Image Turbo merge (Apache-2.0, free for commercial use), finishing a frame in about 8 DiT steps instead of the 30 to 50 a normal diffusion model needs. Runs as a standard Diffusers pipeline or a single merged ComfyUI UNET, on one consumer GPU rather than a rented cluster. Blind preference vs GPT Image 2: 44-16-30 over 90 pairs (internal raters). Large gap ruled out; parity not established. Stronger skin, light, and texture than the bare base; strong on photo, portrait, and mood, and weak on legible text including Hangul and most non-Latin script. ~1.8s end-to-end per image on one g7e.2xlarge (NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB; model resident, queue excluded). Fully loaded unit cost ~1/40 of API list price at measured 50% GPU utilization (owned-hardware cost vs published price; utilization-dependent). Samples below are Redrob Image outputs.
My role
Model selection (Z-Image Turbo open weights), blind preference design, and Studio bake-off ops shared with the client shell.
Purpose
Ship an Apache-2.0 open-weight image model fast enough for interactive bake-offs and competitive enough on preference against a strong hosted baseline, with weights free to use and free for commercial use, and light enough that one consumer GPU is the target rather than the fallback.
Preference study
Blind pairwise preference of Redrob Image vs GPT Image 2 (n=90). Context: frontier-API pricing and 20-180s latency per image. Criterion: product fitness at a defendable unit cost. Outcome set the production image model Studio serves.
Design
- Pairwise vs GPT Image 2; ties permitted
- Rater and experimenter blind to system identity
- 15 prompt categories fixed pre-collection; 6 comparisons each (90 total)
- Two independent rounds; different rater and prompt set per round
- Prompts from the product distribution, not a public benchmark
- Budget: KRW 500,000; three weeks
Round 2 is not a re-rating of round 1. Different rater, different prompts: two independent replications.
Ties: 17.8%. Forcing ties into wins or losses would invent signal.
Experimenter blinding constrained prompt selection, presentation order, and mid-study decisions.
Result
- Redrob Image wins
- 44
- Ties
- 16(17.8%)
- GPT Image 2
- 30
- Ties excluded
- 44/74 = 59.5%95% CI [47.4%, 70.7%] · p = 0.130 · Clopper-Pearson
- Ties split
- 52/90 = 57.8%p = 0.140
- Ties as losses
- 44/90 = 48.9%95% CI [38.2%, 59.7%] · p = 0.916 · Clopper-Pearson
44-16-30 over 90 blind pairs (internal raters). At this n the interval is roughly ±10 points: large gaps ruled out, parity not established. Decision study, not a published claim. Ties-excluded 59.5%, 95% CI [47.4%, 70.7%], p = 0.130: the interval includes 50%. All three tie treatments yield the same non-parity verdict.
Limits
n=90, internal raters, one prompt distribution. Underpowered by design; raters not blind to who built the system under test.
Where it loses
Weakest categories (Redrob Image : tie : GPT Image 2; 6 comparisons each):
- Risograph graphic illustration
- 1 : 0 : 5
- Pixel-art / photoreal compositing
- 2 : 0 : 4
- Graphic design, halftone text overlay
- 2 : 0 : 4
- Vintage technicolor film
- 1 : 1 : 4
Strongest categories (contrast):
- Dramatic portrait
- 5 : 0 : 1
- Dark-mood digital art
- 5 : 1 : 0
- Analog black-and-white portrait
- 4 : 2 : 0
- Double-exposure composite
- 4 : 1 : 1
Photographic and portrait categories favor Redrob Image; graphic, print, and text-bearing categories favor GPT Image 2. This pattern informed routing. It is not established at the cell level.
Omnibus test for category dependence: p = 0.096 (suggestive, not significant). No individual category clears Bonferroni for 15 tests: with 6 comparisons per cell, even 6-0 yields p = 0.031 against a corrected α = 0.0033. The design cannot produce a significant single-category result. That is a sizing flaw, not a model property. Cell patterns may inform routing; they are not established findings.
Post-hoc grouping of the 15 categories into families makes the pattern look decisive. Those groupings were not pre-specified; corresponding p-values are not reported. Categories were fixed in advance; families were not.
Text rendering drove the operational rule. Direct evidence is thin (halftone-text 2-0-4, p = 0.69). Support is directional convergence across adjacent graphic/print cells plus a known structural failure mode for this model class, not a significant cell result.
Operational rule: surfaces that require legible text route to the frontier API.
What I would change
- Six comparisons per category make single-category claims structurally impossible. Minimum useful cell size: 10.
- No rater overlap between rounds: inter-rater reliability uncomputable; rater effects confounded with prompt-set effects. A shared 5-item anchor set would have fixed this at negligible cost.
- Underpowered for equivalence. ±10 points at 80% power needs 155 comparisons; ±5 points, 619. Current data supports at most a ±19-point margin.
- Category families should have been pre-specified. Without that, the family pattern is a hypothesis, not a finding.
- Settling rerun: ~5× current cost (~KRW 2.5M, 3-4 weeks with parallel raters).
Open weights are Apache-2.0 and free for commercial use. Speed came from Z-Image Turbo distillation. The evaluation evidence and the weights are the durable assets.
Decisions
Drop Stable Diffusion for natural-prompt anatomy
Tested Stable Diffusion against Klein 4B and Z-Image Turbo. SD requires a narrow prompt dialect for clean anatomy; ordinary language failed. That dialect was not acceptable.
Reject Klein despite photorealism
Klein 4B was strong on skin and light but broke typography on graphic and print-adjacent work. Photorealism alone was insufficient.
Ship open weights on Z-Image Turbo
Released weights are a Z-Image Turbo merge (Diffusers transformer or single ComfyUI UNET) at about 8 DiT steps. Tuned for more realistic skin, light, and texture than the bare base. Apache-2.0 on the weights themselves, not only the code: free to use, free for commercial use, redistributable.
No custom runtime to learn
It loads through the standard Diffusers pipeline API with from_pretrained, and ships to ComfyUI as one merged UNET. There is no wrapper of ours in the path, nothing to sign up for, and no key to fetch.
Turbo sampling costs you the negative prompt
Eight steps only works without classifier-free guidance (guidance_scale=0, ComfyUI cfg 1), and with guidance off the negative prompt is ignored entirely. Avoidances have to be written into the positive prompt, which is a real constraint rather than a documentation gap.
Keep strengths explicit
Strong on photo, portrait, and mood; weak on legible text, including Hangul and most non-Latin script, and well below the photo results on typography-heavy graphic and print work. Preference study pattern matches that split. Text-bearing surfaces keep a frontier-API escape hatch.
Challenges
Hosted APIs were too slow to iterate
At 20s-3min per image, LoRA and checkpoint sweeps were impractical. Self-hosted ~1.8s end-to-end (same hardware conditions as above) made interactive bake-offs feasible.
Category cells were underpowered
Six comparisons per category cannot establish per-style winners. Shipping used the overall n=90 preference result only. Graphic and print cells failed in the same direction.
Limits
n=90, internal raters, one prompt distribution. Underpowered by design; raters not blind to who built the system under test.
Impact
Measured path with open weights
44-16-30 justified continued development, not a parity claim. Latency supported interactive bake-offs. Unit cost ~1/40 of API list price at measured 50% utilization (owned hardware vs published price; moves with utilization). Weights are Apache-2.0 and free for commercial use.
Distillation, not a proprietary stack
Speed came from base-model distillation on Z-Image Turbo, not a closed serving stack. The evaluation evidence and open weights are the durable assets.
Unit cost
Fully loaded self-hosted cost per image vs GPT Image 2 at $0.04. ~1/40 of API list price at measured 50% GPU utilization. Owned-hardware cost vs published price; not like-for-like; utilization-dependent.
Self-hosted unit cost is dominated by fixed GPU capacity. The figure is undefined without a utilization assumption.
Outputs
System
Redrob Image
1.8s
per image
Open-weight Z-Image Turbo merge (Apache-2.0, free for commercial use); ~1.8s/image on g7e.2xlarge (NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB) after residence, queue excluded; 44-16-30 across 90 blind pairs vs GPT Image 2 (cannot establish parity); stronger skin/light/texture than bare base; strong on photo/portrait/mood; unit cost roughly 1/40th API list at measured utilization, not like-for-like