Redrob Eval

My role

Designed and implemented the TypeScript GEPA harness, the spend-once test API, the export path, the blind preference bracket and the routing labels it produces, and the three-module workbench (Compare, Evolve, Deploy).

Purpose

Given a task, dataset, and quality floor, search for the cheapest configuration that still clears the floor, and collect task-grounded evidence when public Elo is not enough. Model choice is one gene in the GEPA search space; the product is the loop: Compare to pick a model, Evolve under a quality floor, Deploy what you chose, then put the served endpoint back in Compare against the frontier. Reports export as baseline versus evolved (Markdown or JSON).

Method guarantees

• Test is spent once

  Train and validation may be optimized against. Test may not. The reporting endpoint refuses to return test results for a run that optimized against test.

• Relative cost only

  Cost is a percentage of the run baseline, never an absolute currency figure. Self-hosted models have no $/token at all, so their relative cost comes from throughput measured on the GPU host, and FP8 versus bf16 always appears in the result caveats.

• Scores travel with their reasons

  Every metric returns feedback text alongside the number, so a regression can be read rather than guessed at.

• Offline by default, keys server-side

  Datasets are vendored under Apache-compatible licenses and evaluation runs locally. Only provider API calls leave the machine, and provider keys plus GPU host config stay in the gitignored root .env rather than reaching the browser.

Decisions

• GEPA reimplemented in TypeScript

  Rebuilt from the paper (arXiv:2507.19457), not wrapped or ported file by file, with a parity script against the reference implementation. A seed model runs the candidate prompt; a reflect model rewrites it from failure feedback.

• Search space is the configuration, not only the model

  GEPA searches instruction, demonstrations, model choice, script_policy, and frame_policy under a quality floor. Model selection is one gene, not a separate product.

• Compare is the front door, not a leaderboard

  Curated, OpenRouter, direct frontier, and self-hosted vLLM models sit in the same list and run live on a catalog dataset or your own prompts, on text or image. Quality is scored only where reference answers exist; latency, TTFT, and throughput are measured on every run. There is no cost column, because published pricing is never real time.

• One blind bracket per prompt settles what metrics cannot

  Tasks without reference answers go to a single-elimination bracket: two answers at a time with model names hidden, the winner advances, the champion takes the prompt. Non-power-of-two fields pad with byes and a model that errored loses by walkover, so an unusable answer cannot win on presentation.

• Preference votes become routing labels

  Every prompt the fast model won or tied becomes a small label and the rest escalate to the fallback. Those land in the same routing corpus the metric-derived collector fills, so the export and training path stay unchanged while the supervision becomes human instead of metric.

• Deploy closes the loop on hardware you own

  Serve self-hosted models on your own GPU host over SSH with measure, start, health, benchmark, and a resumable terminal, then put the served endpoint back in Compare against the frontier. Benchmark is also what makes a self-hosted relative cost possible at all, since it records measured tokens per second.

• Tokenizer fertility as a budget constraint

  Indic scripts tokenize inefficiently, so the number of demonstrations that fit in a context budget is a function of the script. Reports surface demos_requested against demos_fitted.

• Judge when there is no reference answer

  Custom goal mode takes a rubric and input-only JSONL and scores with an LLM judge, or with checklist QWK when the target is agreement with human graders rather than task accuracy. Tasks without gold answers stay in scope.

Challenges

• Keyword and length routers failed the dual-label check

  Rules keyed on prompt length or keywords agreed with dual-model labels only about 25% of the time on some GSM8K slices. Not usable as a production policy.

• High fertility shrinks demo budget

  When tokenizer fertility is high, demos that were requested do not fit. Reports expose the demos_requested versus demos_fitted gap so the optimizer is not blamed for a context limit.

• Bad labels are unrecoverable downstream

  A learned router cannot fix weak collection UI, pass/fail thresholds, or offline replay. Label quality has to be trustworthy before any routing model is trained. On the labels that mattered, a trained router failed to clear chance and was dropped; the labeling pipeline survived.

Impact

• A public, runnable Apache-2.0 harness

  Clone, set one provider key, and the verification scripts run offline. The artifact is the harness, not a private router checkpoint.

• Baseline-versus-evolved reports that survive review

  Exports with test spent once and cost stated relatively are auditable by someone who did not run them.

• Self-hosted models get an honest cost number

  A model on your own GPU has no published price to quote, so Deploy benchmarks it and Compare prices it against the baseline by measured throughput. That is what lets an owned endpoint sit in the same table as a frontier API without inventing currency.

• Evidence to drop hand-written routing

  The 25% agreement result from dual-model labeling was enough to stop keyword and length rules from shipping as a default production policy.

System

• redrob-eval (public)

  GEPA

  arXiv:2507.19457 / Apache-2.0

  LLM evaluation workbench over instruction, demos, model, and policies under a quality floor. Three modules plus settings: Compare, Evolve, Deploy. Blind preference brackets inside Compare become routing labels.

  

Screens