Skip to content

Fix Japanese tokenizer URL handling - #14029

Open
bhushanasati25 wants to merge 1 commit into
explosion:masterfrom
bhushanasati25:fix/ja-tokenizer-url-match
Open

Fix Japanese tokenizer URL handling#14029
bhushanasati25 wants to merge 1 commit into
explosion:masterfrom
bhushanasati25:fix/ja-tokenizer-url-match

Conversation

@bhushanasati25

@bhushanasati25bhushanasati25 commented Aug 31, 2026

Copy link
Copy Markdown

Fixes#13928.

Description

The Sudachi-backed Japanese tokenizer now protects URL spans before tokenizing the surrounding text, so URLs such as https://example.com/jp stay as one token in Japanese sentences. The tokenizer uses spaCy's shared URL matcher for validation, preserves doc.text_with_ws, and marks the resulting token with the normal lexical LIKE_URL attribute.

Types of change

Bug fix.

Tests

  • python -m pytest -q spacy/tests/lang/ja/test_tokenizer.py -k keeps_urls
  • python -m pytest -q spacy/tests/lang/ja/test_tokenizer.py -k 'not sub_tokens'
  • python -m ruff format spacy/lang/ja/__init__.py spacy/tests/lang/ja/test_tokenizer.py --check
  • python -m ruff check spacy/lang/ja/__init__.py spacy/tests/lang/ja/test_tokenizer.py --select I
  • git diff --check

Note: python -m pytest -q spacy/tests/lang/ja/test_tokenizer.py fails one existing test_ja_tokenizer_sub_tokens expectation with the current sudachidict_core==20260723, where the first two subtoken entries are [[ ]] instead of None. The new URL regression passes.

Checklist

  • I confirm that I have the right to submit this contribution under the project MIT license.
  • I ran the tests, and all new and existing tests passed except for the unrelated SudachiDict-version-sensitive case noted above.
  • My changes don't require a change to the documentation, or if they do, I've added all required information.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Japanese model missing url_match

1 participant

@bhushanasati25