Verified metrics
Generated by scripts/render_metrics.py. Each row names its source file or CI run. Planner and LLM mode are stated in every agent row.
docs/METRICS.md (2026-11-09); references that
only make sense inside the private repository (pointers to internal docs, CI run ids, commit hashes)
are removed, every figure and caveat is kept. A demo run on the home page uses a
small public dataset under the time limits described below; it does not reproduce these benchmark numbers,
which come from larger tasks and longer budgets. The source repository is currently private, so file references below (result JSONs, code) are names only, not links.All 3 paired-difference intervals against FLAML and between the LLM and scripted planners include 0 (10 tasks, each arm run once): no difference between these arms is demonstrated.
A demo run has two limits: a 55 s soft wall after which the agent stops searching and wraps up (review, report), and a 90 s hard limit at which the whole process is killed (the AutoML ceiling is 20 s). Wrap-up and process start come on top of the soft wall, so a finished run can take longer than it.
Agent benchmark v5 (headline; unseen partition, each arm run once)
| Arm | Normalized score [95% CI] | Paired differences | Pass vs linear | Cost / latency | Source |
|---|---|---|---|---|---|
| SCRIPTED planner (no LLM) | 0.987 [0.949, 1.011] | vs linear 0.189 [0.075, 0.315]; vs FLAML -0.013 [-0.051, 0.011] | 7/10 (Wilson 0.40-0.89) | latency p50 125s / p95 199s per task; 0 LLM requests, 0 tokens, $0 | agent_v5_scripted.json |
| REAL-LLM planner v2 | 0.989 [0.950, 1.012] | vs linear 0.190 [0.076, 0.316]; vs FLAML -0.011 [-0.050, 0.012] | 8/10 (Wilson 0.49-0.94) | latency p50 135s / p95 226s per task; 130 LLM requests, 19562 tokens, $0 | agent_v5_llm.json |
Linear baseline normalized mean 0.798 [0.680, 0.908] (n=10); FLAML = 1 by definition. Source: baselines_v5.json. Paired LLM minus scripted: 0.001 [-0.001, 0.004] (n=10).
Leakage detector
- eval4 (unseen planted partition, 30 variants): recall 0.967 (bootstrap 95% [0.9, 1.0]), clean-twin false alarms 0.000 (Wilson 95% [0.0, 0.1135]). Source:
leakage_v3.json.
Injection defence
- Real LLM in the loop (injection_llm_v2): attacker action proposed 0/49, unsafe actions executed 0/49. Source:
injection_llm_v2.json. - Evasion DEV (
injection_llm_v3.json; burned, the scanner was tuned on it): attacker actions 0/8, scanner flagged 8/8. - FRESH 14-payload held-out evasion set (hash-committed before the run, run once): attacker actions proposed 0/14 (Wilson 95% upper bound 0.215), unsafe executed 0, 1 non-train/finish proposal (a benign
read_file train.csv, guard allowed). The scanner flagged only 1/14 of the fresh payloads, so the model's own behaviour and the structural guard carry this result. - 5 full agent runs with injected column names (real planner v2 + critic v3): 5 finished, unsafe tools executed 0, payload reached the critic 0 times (by design the planner never sees column names, so this is a null-exposure check, not an attack); critic flagged 3/5 for ordinary quality reasons (rules flagged 2). Source:
injected_runs_v1.json.
Critic (advisory)
- v5 runs, critic v2 vs rule reviewer: agreement 18/20 (Wilson 95% 0.70-0.97); confusion {'rule=APPROVE|critic=APPROVE': 18, 'rule=REQUEST_CHANGES|critic=APPROVE': 2}; 14 requests. Source:
critic_v5.json. - Dev calibration (mutated states): see
critic_calibration.json. - FRESH planted set (critic v3, schema-enforced answers to failed rules): recall 10/15 (Wilson 95% 0.42-0.85); false alarms 1/19 clean runs (Wilson 95% 0.01-0.25); 4 clean items not run (40-request cap). By type: {'cv_instability': '1/2', 'gap_beyond_family_range': '2/3', 'partial_leakage_check': '2/2', 'rule_fail_baseline_margin': '0/3', 'tiny_test': '2/2', 'unresolved_leakage': '3/3'}. Source:
critic_v6.json, setbenchmarks/critic_set_v6.json(sha2566595774668473af9, committed before the first call). - The v5 agreement above (18/20) is burned and post-hoc: the critic was tuned after reading v5-era failures.
- FRESH planted set v2 (53 items, hash
bfdcaf09a5aa0346, failing-margin items now unambiguous; a DIFFERENT set from v1, not paired): completed run recall 18/23 (Wilson 95% 0.58-0.90), false alarms 0/30 (0.00-0.11). First pass (8 items hit the gateway token budget) was 15/20 and 0/25; the 8 were completed later with the same prompt (labelled completion). Failing-margin items 0/4: the model dissents with numbers, a documented weakness. Source:critic_v2_eval.json.
E2B sandbox backend
- 20/20 sequential execs (default template, internet off): total p50 1.87 s / p95 2.67 s (create p50 0.38 s, upload 0.56 s, exec 0.34 s, download 0.55 s); n=20, one machine, an order-of-magnitude figure. Live suite: 8 passed, 0 failed, 0 xfailed, 2 skipped (the 2 skips need a custom template with pyarrow, not built). Source:
e2b_v1.json.
Demo
- LIVE (host: Render free plan, 512 MB, 0.1 vCPU): https://junior-ml-dev-demo.onrender.com . Verified: health 200, landing 200, admin without auth 401, upload attempt 403, deploy path 404, reports [200, 200, 200]. Sample run (diabetes) p50 49.8 s over 3 runs ([49.8, 50.6, 49.0]). Cold start after >15 min idle: health 32.7 s after >15 min idle; first sample run 105.8 s end to end (includes ~45-60 s warm-worker boot); status done. Local peak process-tree RSS of a sample run: 309-327 MB local (Windows psutil process tree, FLAML 15 s); web process alone 63 MB idle. Source:
demo_render_v1.json. - Cut to fit the free plan: FLAML off; leakage detectors skipped (recorded as a PARTIAL check); hard wall 60 s, soft wall 35 s; web process no longer imports the full API (225 MB -> 63 MB); a warm fork server loads pandas/scikit-learn/lightgbm once (they took 44 s per run on the 0.1 vCPU).
- Static precomputed site (no code runs there): https://giri-harsh.github.io/junior-ml-dev-demo/ .
Infrastructure (GitHub Actions evidence)
- latest : startup_failure (0 s)
- latest ci: startup_failure (2 s)
- latest infra: success (459 s)
- latest load: success (203 s)
- latest web: startup_failure (1 s)
- runs sampled: 30 most recent, 24 success
Load test
| Users | RPS | p50 ms | p95 ms | p99 ms | error rate |
|---|---|---|---|---|---|
| 5 | 36.2 | 140 | 210 | 240 | 0.000 |
| 25 | 75.1 | 26 | 1500 | 1600 | 0.000 |
| 100 | 65.7 | 140 | 6700 | 9600 | 0.178 |
Source: load_v1.json (CI run); hosted-runner numbers, not a capacity claim.