xsl
67d9754823
tests/golden：JSON 用例 + 阈值配置 + runner（规则 pass_rate=1.0、检索 0.8 起步）+ pytest 壳；规则/检索两子集种子，Agent/报告子集占位（T-P2-11）。 实现： - runner.py 为普通模块（不依赖 pytest）：加载/执行/指标（pass_rate、 first_hit_in_top3）/逐用例差异明细；test_golden.py 只是薄壳（marker=golden， 无 MEIDA_RUN_INTEGRATION 时整体 skip）。用例与 expect 的未知键一律判 fail（封闭词汇表，防断言静默失效）。 - 规则子集 13 例（rules_cases.json，走 packages.rules DSL/evaluate）： 含 3 个硬禁忌场景（无条件合规硬拦、正式场合硬拦、复合优先级链 100硬>95禁忌>90体型>50趋势）+ 禁忌压制/维度隔离/多规则叠加/温度边界/ 缺字段静默/include 子集语义等。 - 检索子集 13 例（retrieval_cases.json，DeterministicEmbedder+hybrid_search）： 含 7 个 must_miss 红线场景（禁忌体型/draft/archived/场景外/类型外/组合红线）。 种子实体以 golden: 前缀命名，跑前先删（幂等重跑）；跑后也删——CI 防线 5c 与集成测试共用长活库，留下假实体会破坏其它用例对结果列表的精确断言。 确定性策略：query=目标 chunk 原文（余弦距离 0 必排第 1）+ 每例独立 marker token（gqNN）隔离关键词路；检索参数（1024 维/top_k 50/默认 RRF 权重）钉死在 runner，不读 env。 - thresholds.json：规则 pass_rate_min=1.0（硬结论子集一条不许退）； 检索 pass_rate_min=0.8 + first_hit_in_top3_min=0.8 起步。 - README.md：四子集 330 例规划、两格式 spec、加用例流程、阈值口径与 “为什么规则必须 1.0”。 测试： - MEIDA_RUN_INTEGRATION=1 pytest -m golden --no-cov：2 passed（规则 13/13=1.0，检索 13/13=1.0、first_hit_in_top3=13/13）；连续重跑幂等， 跑后库内 golden:% 残留为 0；不设 env 时 2 skipped；基线单测 473 passed、 集成 38 passed 不受影响；ruff check/format、lint-imports 全绿。 - 阻断语义自证（已改回，提交里只有正确版本）：临时把规则用例 user-taboo-overrides-trend-recommendation 的 must_avoid 改成不存在的 取值并把 hard_blocks_nonempty 翻转为 true → 规则子集 12/13=0.9231 <1.0 fail，明细输出“must_avoid 未命中：期望 top[length=九分裤脚]，实际 avoid=[top[length=[短款, 露脐]]]”与“hard_blocks_nonempty 期望 True， 实际 False（硬拦截规则：无）”；再临时破坏 3 条检索用例（2 条 must_hit 换成不存在的实体名、1 条 must_miss 加入本应命中的实体）→ 检索子集 10/13=0.7692<0.8 fail，明细含“must_miss 红线违规：golden:r12-good 出现在第 1 位——硬过滤零漏被破坏”。验证完三条全部改回，复跑全绿。 （注：检索阈值 0.8 起步，单条退化 0.92 尚不拦截、≥3 条即拦截，符合 种子期口径；规则阈值 1.0 单条即拦截。） 偏差说明：无
349 lines
9.5 KiB
JSON
349 lines
9.5 KiB
JSON
[
|
|
{
|
|
"name": "exact-text-match-ranks-first",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r01-exact",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq01 crisp white cotton shirt for office wear"]
|
|
}
|
|
],
|
|
"query": "gq01 crisp white cotton shirt for office wear",
|
|
"user_ctx": {},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r01-exact"],
|
|
"must_miss_names": [],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "taboo-body-shape-entity-never-leaks",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r02-taboo",
|
|
"attrs": {"occasions": ["wedding"], "avoid_body_shapes": ["apple"]},
|
|
"status": "published",
|
|
"chunks": ["gq02 pear body dress styling guide"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r02-good",
|
|
"attrs": {"occasions": ["wedding"]},
|
|
"status": "published",
|
|
"chunks": ["gq02 pear body dress styling guide for apple shapes"]
|
|
}
|
|
],
|
|
"query": "gq02 pear body dress styling guide for apple shapes",
|
|
"user_ctx": {"body_shape": "apple", "occasion": "wedding", "entity_types": ["item"]},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r02-good"],
|
|
"must_miss_names": ["golden:r02-taboo"],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "draft-entity-never-leaks-even-with-near-text",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r03-draft",
|
|
"attrs": {},
|
|
"status": "draft",
|
|
"chunks": ["gq03 sequin party clutch styling notes"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r03-live",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq03 sequin party clutch styling notes published edition"]
|
|
}
|
|
],
|
|
"query": "gq03 sequin party clutch styling notes published edition",
|
|
"user_ctx": {},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r03-live"],
|
|
"must_miss_names": ["golden:r03-draft"],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "archived-entity-never-leaks",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r04-archived",
|
|
"attrs": {},
|
|
"status": "archived",
|
|
"chunks": ["gq04 vintage brooch care instructions"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r04-live",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq04 vintage brooch care instructions revised"]
|
|
}
|
|
],
|
|
"query": "gq04 vintage brooch care instructions revised",
|
|
"user_ctx": {},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r04-live"],
|
|
"must_miss_names": ["golden:r04-archived"],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "undeclared-occasion-entity-excluded-under-filter",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r05-declared",
|
|
"attrs": {"occasions": ["interview"]},
|
|
"status": "published",
|
|
"chunks": ["gq05 navy blazer pairing guide"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r05-undeclared",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq05 navy blazer pairing guide extra tips"]
|
|
}
|
|
],
|
|
"query": "gq05 navy blazer pairing guide",
|
|
"user_ctx": {"occasion": "interview"},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r05-declared"],
|
|
"must_miss_names": ["golden:r05-undeclared"],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "entity-type-filter-restricts-recall",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "style",
|
|
"name": "golden:r06-style",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq06 monochrome layering theory deep dive"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r06-item",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq06 monochrome layering theory applied outfit"]
|
|
}
|
|
],
|
|
"query": "gq06 monochrome layering theory applied outfit",
|
|
"user_ctx": {"entity_types": ["item"]},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r06-item"],
|
|
"must_miss_names": ["golden:r06-style"],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "multi-chunk-entity-hit-on-second-chunk",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r07-multi",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": [
|
|
"gq07 wool coat winter intro",
|
|
"gq07 wool coat collar and lapel details",
|
|
"gq07 wool coat care and storage"
|
|
]
|
|
}
|
|
],
|
|
"query": "gq07 wool coat collar and lapel details",
|
|
"user_ctx": {},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r07-multi"],
|
|
"must_miss_names": [],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "relevant-chunk-outranks-unrelated-sibling",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r08-relevant",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq08 breathable linen shirt summer comfort"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r08-noise",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq08 leather boots winter warm"]
|
|
}
|
|
],
|
|
"query": "gq08 breathable linen shirt summer comfort",
|
|
"user_ctx": {},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r08-relevant"],
|
|
"must_miss_names": [],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "duplicate-content-first-hit-still-top3",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r09-first",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq09 silver pleated midi skirt styling"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r09-second",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq09 silver pleated midi skirt styling"]
|
|
}
|
|
],
|
|
"query": "gq09 silver pleated midi skirt styling",
|
|
"user_ctx": {},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r09-second"],
|
|
"must_miss_names": [],
|
|
"first_hit_in": "top3"
|
|
}
|
|
},
|
|
{
|
|
"name": "topic-siblings-all-recalled",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r10-a",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq10 wide brim hat sun protection notes"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r10-b",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq10 wide brim hat travel packing tips"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r10-c",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq10 wide brim hat material comparison"]
|
|
}
|
|
],
|
|
"query": "gq10 wide brim hat sun protection notes",
|
|
"user_ctx": {},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r10-a", "golden:r10-b", "golden:r10-c"],
|
|
"must_miss_names": [],
|
|
"first_hit_in": "top3"
|
|
}
|
|
},
|
|
{
|
|
"name": "cjk-exact-vector-match",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r11-cjk",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq11 梨形身材 高腰线 连衣裙 穿搭要点"]
|
|
}
|
|
],
|
|
"query": "gq11 梨形身材 高腰线 连衣裙 穿搭要点",
|
|
"user_ctx": {},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r11-cjk"],
|
|
"must_miss_names": [],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "combined-red-line-taboo-draft-archived",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r12-taboo",
|
|
"attrs": {"avoid_body_shapes": ["pear"]},
|
|
"status": "published",
|
|
"chunks": ["gq12 column line coat structure guide"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r12-draft",
|
|
"attrs": {},
|
|
"status": "draft",
|
|
"chunks": ["gq12 column line coat structure guide draft"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r12-archived",
|
|
"attrs": {},
|
|
"status": "archived",
|
|
"chunks": ["gq12 column line coat structure guide archive"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r12-good",
|
|
"attrs": {},
|
|
"status": "published",
|
|
"chunks": ["gq12 column line coat structure guide final"]
|
|
}
|
|
],
|
|
"query": "gq12 column line coat structure guide final",
|
|
"user_ctx": {"body_shape": "pear"},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r12-good"],
|
|
"must_miss_names": [
|
|
"golden:r12-taboo",
|
|
"golden:r12-draft",
|
|
"golden:r12-archived"
|
|
],
|
|
"first_hit_in": "top1"
|
|
}
|
|
},
|
|
{
|
|
"name": "occasion-positive-declared-hit",
|
|
"seed_entities": [
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r13-gala",
|
|
"attrs": {"occasions": ["gala", "party"]},
|
|
"status": "published",
|
|
"chunks": ["gq13 velvet wrap dress evening notes"]
|
|
},
|
|
{
|
|
"entity_type": "item",
|
|
"name": "golden:r13-office",
|
|
"attrs": {"occasions": ["office"]},
|
|
"status": "published",
|
|
"chunks": ["gq13 velvet wrap dress evening notes office edition"]
|
|
}
|
|
],
|
|
"query": "gq13 velvet wrap dress evening notes",
|
|
"user_ctx": {"occasion": "gala"},
|
|
"expect": {
|
|
"must_hit_names": ["golden:r13-gala"],
|
|
"must_miss_names": ["golden:r13-office"],
|
|
"first_hit_in": "top1"
|
|
}
|
|
}
|
|
]
|