[linhongyu510]

enable_truncation() resets the direction to right unless it is explicitly passed. Save the configured direction from the tokenizer before re-enabling truncation at MAX_LENGTH, then pass it through. Same bug class as #801 for the ColBERT loader.

7 regression tests covering left/right direction, tail/head truncation with 600 tokens, no-truncation default, no-padding, and save/reload roundtrip.

All Submissions:

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

• Does your submission pass the existing tests?
• Have you added tests for your feature?
• Have you installed pre-commit with pip3 install pre-commit and set up hooks with pre-commit install?

New models submission:

• Have you added an explanation of why it's important to include this model?
• Have you added tests for the new model? Were canonical values for tests computed via the original model?
• Have you added the code snippet for how canonical values were computed?
• Have you successfully ran tests with your changes locally?

[coderabbitai]

Walkthrough

load_static_tokenizer now preserves the tokenizer’s configured truncation direction when applying the 512-token limit. If no direction is configured, it uses right truncation. New regression tests cover truncation direction, retained tokens, padding, and save-and-reload behavior.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Merge Risk: Minimal · up to 73cbc

No actionable merge-blocking risk is identified.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly and concisely identifies the main change: preserving truncation direction in the static tokenizer loader.
Description check		The description directly explains the truncation-direction fix and the related regression tests, so it is relevant to the changeset.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 2 functions across 2 files.

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}