junior-ml-dev

Verified metrics

Generated by scripts/render_metrics.py. Each row names its source file or CI run. Planner and LLM mode are stated in every agent row.

Rendered from docs/METRICS.md (2026-11-09); references that only make sense inside the private repository (pointers to internal docs, CI run ids, commit hashes) are removed, every figure and caveat is kept. A demo run on the home page uses a small public dataset under the time limits described below; it does not reproduce these benchmark numbers, which come from larger tasks and longer budgets. The source repository is currently private, so file references below (result JSONs, code) are names only, not links.

All 3 paired-difference intervals against FLAML and between the LLM and scripted planners include 0 (10 tasks, each arm run once): no difference between these arms is demonstrated.

A demo run has two limits: a 55 s soft wall after which the agent stops searching and wraps up (review, report), and a 90 s hard limit at which the whole process is killed (the AutoML ceiling is 20 s). Wrap-up and process start come on top of the soft wall, so a finished run can take longer than it.

Agent benchmark v5 (headline; unseen partition, each arm run once)

ArmNormalized score [95% CI]Paired differencesPass vs linearCost / latencySource
SCRIPTED planner (no LLM)0.987 [0.949, 1.011]vs linear 0.189 [0.075, 0.315]; vs FLAML -0.013 [-0.051, 0.011]7/10 (Wilson 0.40-0.89)latency p50 125s / p95 199s per task; 0 LLM requests, 0 tokens, $0agent_v5_scripted.json
REAL-LLM planner v20.989 [0.950, 1.012]vs linear 0.190 [0.076, 0.316]; vs FLAML -0.011 [-0.050, 0.012]8/10 (Wilson 0.49-0.94)latency p50 135s / p95 226s per task; 130 LLM requests, 19562 tokens, $0agent_v5_llm.json

Linear baseline normalized mean 0.798 [0.680, 0.908] (n=10); FLAML = 1 by definition. Source: baselines_v5.json. Paired LLM minus scripted: 0.001 [-0.001, 0.004] (n=10).

Leakage detector

Injection defence

Critic (advisory)

E2B sandbox backend

Demo

Infrastructure (GitHub Actions evidence)

Load test

UsersRPSp50 msp95 msp99 mserror rate
536.21402102400.000
2575.126150016000.000
10065.7140670096000.178

Source: load_v1.json (CI run); hosted-runner numbers, not a capacity claim.