Repository navigation
cdc: update storage sink (#21221) - #21998
ti-chi-bot wants to merge 1 commit into
Conversation
ti-chi-bot
added
area/ticdc
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
📝 Walkthrough
Merge Risk: 🟡 Moderate · up to
|
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Description check |  |
描述基本完整。它说明这是 #21221 的自动 cherry-pick，填写了适用版本（master 与 v8.5），并给出了相关 TiCDC PR 链接。但“What is changed, added or deleted?”一节为空，未说明本次文档改动的内容，即 Storage Sink 文件命名变更及下游消费顺序说明。该部分为必填项。 |
在“What is changed, added or deleted? (Required)”下补充以下内容：1）在 v8.5 新架构启用表级任务拆分（scheduler.enable-table-across-nodes）时，Storage Sink 数据文件名由 CDC{num}.{extension} 改为 CDC_{uuid}{num}.{extension}，Index 文件名由 CDC.index 改为 CDC{uuid}.index；2）ticdc-arc… |
-
Check name Status Explanation Title check 标题“cdc: update storage sink ( #21221)”描述了主要变更对象，即 TiCDC 的 Storage Sink。它与变更集相关，但未说明具体改动（文件命名从 CDC{num} 改为 CDC_{uuid}_{num}，以及表级任务拆分的相关行为）。该标题较为宽泛，但不是模糊或通用的表述，故判定为通过。Docstring Coverage No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Full details: Docstring Coverage
-
Explanation
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (3 skipped: 3 unsupported.)
-
Full details: Description check
-
Resolution
在“What is changed, added or deleted? (Required)”下补充以下内容：1）在 v8.5 新架构启用表级任务拆分（scheduler.enable-table-across-nodes）时，Storage Sink 数据文件名由 CDC{num}.{extension} 改为 CDC_{uuid}{num}.{extension}，Index 文件名由 CDC.index 改为 CDC{uuid}.index；2）ticdc-architecture.md 与 ticdc-storage-consumer-dev-guide.md 中新增的消费说明，包括读取顺序与按 commit-ts 排序；3）release-8.5.5.md 中的行为变更条目。另请说明“This PR is translated from”一项是否需要填写。
✨ Finishing Touches: 💡 1
-
🛠️ Fix failing CI checks: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Source mode: Auto
- Repository: pingcap/docs-cn
- Branch: release-8.5
- File: .coderabbit.yaml
- Commit: 6adbbbc
- Layers: Repository YAML
- Review profile: ASSERTIVE
- Plan: Advanced
- Run ID:
f4c39bec-996b-4560-9098-418124fafd2c
📒 Files selected for processing: 3
releases/release-8.5.5.mdticdc/ticdc-architecture.mdticdc/ticdc-storage-consumer-dev-guide.md
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.
|
|
||
| 在处理文件时，可能会出现某个数据文件尚未完全写入就被下游读取的情况，导致这部分数据未被成功读取。为了避免这种情况，编写下游消费程序时，请按照以下顺序读取数据： | ||
|
|
||
| 1. 读取 `{schema}/{table}/{table-version-separator}/` 目录下的 Index 文件，获取各个 Index 文件中记录的文件名。未开启表级任务拆分时，读取 `meta/CDC.index`；在 TiCDC 新架构下开启表级任务拆分（`scheduler.enable-table-across-nodes = true`）后，分别读取各个子同步任务对应的 `meta/CDC_{uuid}.index` 文件。 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
补全 Index 文件的完整路径。
当前步骤遗漏了分区目录和日期目录层级。当这些分隔符非空时，消费者可能扫描错误目录，无法发现 Index 文件。官方 Storage Sink 路径定义也包含这些目录层级。 (docs.pingcap.com)
可直接应用的替换
--- "a/ticdc/ticdc-storage-consumer-dev-guide.md"
+++ "b/ticdc/ticdc-storage-consumer-dev-guide.md"
@@ -185,7 +185,7 @@
在处理文件时，可能会出现某个数据文件尚未完全写入就被下游读取的情况，导致这部分数据未被成功读取。为了避免这种情况，编写下游消费程序时，请按照以下顺序读取数据：
-1. 读取 `{schema}/{table}/{table-version-separator}/` 目录下的 Index 文件，获取各个 Index 文件中记录的文件名。未开启表级任务拆分时，读取 `meta/CDC.index`；在 TiCDC 新架构下开启表级任务拆分（`scheduler.enable-table-across-nodes = true`）后，分别读取各个子同步任务对应的 `meta/CDC_{uuid}.index` 文件。
+1. 读取 `{schema}/{table}/{table-version-separator}/{partition-separator}/{date-separator}/meta/` 目录下的 Index 文件，获取各个 Index 文件中记录的文件名。未开启表级任务拆分时，读取 `CDC.index`；在 TiCDC 新架构下开启表级任务拆分（`scheduler.enable-table-across-nodes = true`）后，分别读取各个子同步任务对应的 `CDC_{uuid}.index` 文件。
2. 对于 [TiCDC 新架构](/ticdc/ticdc-architecture.md)，针对每个 `CDC_{uuid}.index`，依次读取与该 Index 文件具有相同 `uuid`，且文件序号小于或等于该 Index 文件记录序号的数据文件。对于 [TiCDC 老架构](/ticdc/ticdc-classic-architecture.md)，依次读取文件序号小于该文件名中序号的文件。
> **注意：**📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| 1. 读取 `{schema}/{table}/{table-version-separator}/` 目录下的 Index 文件，获取各个 Index 文件中记录的文件名。未开启表级任务拆分时，读取 `meta/CDC.index`；在 TiCDC 新架构下开启表级任务拆分（`scheduler.enable-table-across-nodes = true`）后，分别读取各个子同步任务对应的 `meta/CDC_{uuid}.index` 文件。 | |
| 1. 读取 `{schema}/{table}/{table-version-separator}/{partition-separator}/{date-separator}/meta/` 目录下的 Index 文件，获取各个 Index 文件中记录的文件名。未开启表级任务拆分时，读取 `CDC.index`；在 TiCDC 新架构下开启表级任务拆分（`scheduler.enable-table-across-nodes = true`）后，分别读取各个子同步任务对应的 `CDC_{uuid}.index` 文件。 |
2 participants
This is an automated cherry-pick of #21221
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions (in Chinese).
What is the related PR or file link(s)?
Do your changes match any of the following descriptions?
Summary by CodeRabbit
CDC_{uuid}_{num}.{extension}，索引文件名变更为CDC_{uuid}.index。下游消费时需根据各子任务的索引文件匹配数据文件，并按commit-ts排序处理 DML。