Skip to content

Japanese TN fix and improvement - #444

Open
folivoramanh wants to merge 5 commits into
NVIDIA:staging/ja_tn_v1from
folivoramanh:ja_tn
Open

Japanese TN fix and improvement#444
folivoramanh wants to merge 5 commits into
NVIDIA:staging/ja_tn_v1from
folivoramanh:ja_tn

Conversation

@folivoramanh

@folivoramanhfolivoramanh commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here.
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files?
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. See an example here.
  • Remove import guards (try import: ... except: ...) if not already done.
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo).
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py.

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

@folivoramanh
Japanese TN fix and improvement
ffa2255
Signed-off-by: Mai Anh <palasek182@gmail.com>
@folivoramanh
folivoramanh requested a review from mgrafuJuly 2, 2026 18:40
@folivoramanhfolivoramanh self-assigned this Jul 2, 2026
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/data/latin/letters.tsv
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/data/money/__init__.py
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/data/money/quantity.tsv Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/data/roman/key_ordinal.tsv Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/data/roman/preserve.tsv Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/verbalizers/postprocessor.py
@mgrafu

mgrafu commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

please update jenkinsfile

mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadtests/nemo_text_processing/ja/test_sparrowhawk_normalization.sh
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/taggers/electronic.py Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/taggers/address.py
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/inverse_text_normalization/ja/data/numbers/zero.tsv
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/data/whitelist_title.tsv
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/taggers/address.py Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/data/numbers/zero_decimal.tsv
@@ -0,0 +1 @@
0 零

@mgrafumgrafuJul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this different from zero.tsv and both from the ITN zero?

@folivoramanhfolivoramanhJul 17, 2026

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

零 is kanji form, and in case of 0.5, 零 is preferred for shorten written, while normal cardinal can use direct kana form

mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/taggers/decimal.py Outdated
graph_integer = (
pynutil.insert('integer_part: \"')
+ (
pynini.cross("0", "零")

@mgrafumgrafuJul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wouldn't this be zero_decimal?

mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/taggers/punctuation.py Outdated
def __init__(self, deterministic: bool = True):
super().__init__(name="punctuation", kind="classify", deterministic=deterministic)
s = "!#$%&'()*+,-./:;<=>?@^_`{|}。，；：《》“”·~【】！？、‘’.<>-——_、。.「」『』‘`／・；’”“”‷･〔〕々〃ゝゞヽ〲〱〳〴〵ヾ〆，~"
s = "!#$%&'()*+,-./:;<=>?@^_`{|}~"

@mgrafumgrafuJul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we sure we don't need this punctuation?

mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/taggers/serial.py Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/taggers/serial.py Outdated
mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadnemo_text_processing/text_normalization/ja/taggers/time.py Outdated
division = pynini.string_file(get_abs_path("data/time/division.tsv"))

division_component = pynutil.insert("suffix: \"") + division + pynutil.insert("\"")
hour_number = pynutil.add_weight(pynini.cross("0", "零"), -0.1) | graph_cardinal

@mgrafumgrafuJul 9, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again looks like zero_decimal?

mgrafu
mgrafu reviewed Jul 9, 2026
View reviewed changes
Comment threadtests/nemo_text_processing/ja/test_telephone.py Outdated
folivoramanhand others added 3 commits July 17, 2026 08:39
@folivoramanh
update ja tn coding convention
9084c74
Signed-off-by: Mai Anh <palasek182@gmail.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
78dc958
for more information, see https://pre-commit.ci
@folivoramanh
update tsv formating
f2cc489
Signed-off-by: Mai Anh <palasek182@gmail.com>
@github-actions

github-actionsBot commented Aug 1, 2026

Copy link
Copy Markdown

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@github-actions

github-actionsBot commented Aug 9, 2026

Copy link
Copy Markdown

This PR was closed because it has been inactive for 7 days since being marked as stale.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mgrafumgrafumgrafu left review comments

Assignees

@folivoramanhfolivoramanh

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@folivoramanh@mgrafu