Skip to content

fix(storage): repair skills_fts indexing on legacy databases - #2504

Open
zichen0116 wants to merge 4 commits into
MemTensor:mainfrom
zichen0116:fix/skills-fts-migration-backfill
Open

zichen0116 wants to merge 4 commits into
MemTensor:mainfrom
zichen0116:fix/skills-fts-migration-backfill

Conversation

@zichen0116

Copy link
Copy Markdown

Description

Fixes #2500: on databases created before the skills_fts FTS5 sync triggers landed in 001-initial.sql, the skills table has no skills_fts_ai / _ad / _au triggers — 001 never re-runs on existing databases, and unlike policies FTS (which got repair migration 009-policies-fts.sql) no later migration ever added them. Result: every INSERT into skills (including via importBundle()) leaves skills_fts empty, so the keyword channel in skill retrieval (WHERE skills_fts MATCH ... in core/retrieval/tier1-skill.ts) never matches — exactly the reported symptom (policies/world-model FTS populated, skills FTS at 0).

Implementation approach:

  • apps/memos-local-plugin/core/storage/migrations/020-skills-fts.sql: creates the skills_fts virtual table and the three sync triggers with IF NOT EXISTS, then backfills SELECT id, name, invocation_guide FROM skills WHERE id NOT IN (SELECT skill_id FROM skills_fts) — same shape and repair pattern as 009.
  • apps/memos-local-plugin/core/storage/migrator.ts: guard 020 with tableExists(db, "skills") so partial-schema test harnesses don't fail boot (same guard style as 009/012/018).

The migration is a no-op on fresh databases (001 already created everything there).

Related Issue (Required): Fixes #2500

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Unit Test — new regression test tests/unit/storage/skills-fts-migration.test.ts simulates a legacy DB (drops the triggers, empties FTS), reproduces the empty index, then applies the migration SQL: triggers restored, existing rows backfilled, trigram MATCH works, replay is idempotent, and subsequent insert/update/delete stay in sync; plus a fresh-DB no-op case.
  • Full run: npx vitest run tests/unit/storage/skills-fts-migration.test.ts tests/unit/storage/migrator.test.ts tests/unit/storage/fts-keyword.test.ts tests/unit/storage/keyword.test.ts → 36/36 passed; npm run lint (tsc --noEmit) clean.

Checklist

  • I have performed a self-review of my own code | 我已自行检查了自己的代码
  • I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • I have created related documentation issue/PR in MemOS-Docs (if applicable) | N/A — internal schema repair, no user-facing behavior change beyond the bug being fixed
  • I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)

Copilot AI balanced review requested due to automatic review settings October 15, 2026 19:13
@Memtensor-AI Memtensor-AI added area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Oct 15, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@Memtensor-AI

Memtensor-AI commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2504
Task: da231378aac06e41
Base: main
Head: fix/skills-fts-migration-backfill

🔍 OpenCodeReview found 1 issue(s) in this PR.


1. apps/memos-local-plugin/core/storage/migrations/020-skills-fts.sql (L3-L9)

The final INSERT INTO skills_fts backfill runs unconditionally after DROP TABLE IF EXISTS skills_fts + CREATE VIRTUAL TABLE, so on a fresh/already-correct database the entire FTS index is always rebuilt from scratch on every replay of this migration. While the test at line 51-54 verifies idempotency (the content is the same after a second run), the DROP TABLE + recreate + full re-insert is still executed every time the migration is re-applied.

More importantly, the DROP TABLE IF EXISTS skills_fts will also cascade-drop all FTS shadow tables (skills_fts_data, skills_fts_idx, skills_fts_content, etc.) and the three triggers are dropped separately before that. There is a subtle ordering risk: if the script is run outside a transaction (e.g., during a crash between the DROP TABLE and CREATE VIRTUAL TABLE statements), the skills_fts table will be left missing entirely. The migrator wraps v20 with execIfTable but does NOT wrap it in a transaction — consider ensuring the entire script is atomic.

Suggestion: Wrap the DDL+DML in an explicit BEGIN/COMMIT block within the SQL file itself, or confirm that execIfTable / the caller always executes this inside a transaction.

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Environment preparation failed before any gating tests executed. Failed scopes: memos_local_plugin
Branch: fix/skills-fts-migration-backfill

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Environment preparation failed before any gating tests executed. Failed scopes: memos_local_plugin
Branch: fix/skills-fts-migration-backfill

…nitions

Drop and rebuild the FTS table/triggers instead of IF NOT EXISTS so a
drifted legacy definition (missing column, wrong tokenizer) cannot
silently survive and break trigger creation. Restrict the update
trigger to indexed columns to avoid write amplification on
usage_count/vec/status bumps.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:plugin OpenClaw & Hermes status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

importBundle 导入 skills 时不会写 skills_fts,技能的关键词检索永远是空的

4 participants