Uh oh!
There was an error while loading. Please reload this page.
Ar TN/TN - #454
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| ) | ||
| self.final_graph_wo_negative = optional_graph_negative + self.final_graph_wo_sign | ||
| final_graph = self.add_tokens(self.final_graph_wo_negative) |
There was a problem hiding this comment.
do we need to remove coverage for negatives?
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
…d fraction homograph Cardinal (TN/ITN): - Expand to millions with place-value 3-digit period decomposition - Orthography: مئة (drop silent alif), مئتين, separated compound hundreds (ثلاث مئة), مئتي construct-state before counted nouns - Use genitive tens (عشرين) and اثنين by default - Fix counted-noun (تمييز) agreement: trailing 3-10 -> plural (110000 -> مئة وعشرة آلاف), otherwise singular; applied to آلاف and ملايين Fraction (TN/ITN): - Resolve homograph collision between dual fractions and cardinals by using construct-state dual denominators without final nun (سبعي=2/7 vs سبعين=70); both directions now round-trip losslessly - Verbalizer accepts اثنين numerator Tests: - Fix pre-existing NameError (RUN_AUDIO_BASED_TESTS) in ar/test_measure.py - Regenerate cardinal/fraction/decimal/money/measure test data (TN+ITN) to the new canonical spelling; decimal/money/measure grammars unchanged - Full ar suite: 306 passed Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
TN (text_normalization/ar):
- Add time tagger + verbalizer for HH:MM, HH:MM:SS, and H <suffix> inputs
- Formal/regular style: hour as feminine ordinal (24h; midnight 0 -> الثانية عشرة),
minutes/seconds as <number> دقيقة/دقائق
- Counted-noun agreement:
* teens via 13_19 data (11 -> إحدى عشرة, 15 -> خمس عشرة)
* 3-10 plural (خمس دقائق), 1 -> دقيقة واحدة, 2 -> دقيقتان
* 21-59 ones digit feminized by polarity (خمس/ثمان/إحدى/اثنتان ...)
* minutes/seconds number is nominative (عشرون/ثلاثون/أربعون/خمسون)
- Constrain suffix verbalization to suffix.tsv values (required for a clean inverse)
- Fix suffix.tsv tab separators
- Wire TimeFst into tokenize_and_classify.py and verbalize.py
- Add data: time/{suffix,time_zone}.tsv, ordinal/ordinals.tsv, number/{3_10mas,13_19}.tsv
ITN (inverse_text_normalization/ar):
- Add time tagger (inverts the TN time verbalizer) + verbalizer (-> HH:MM[:SS])
- Wire into tokenize_and_classify.py and verbalize.py
Tests:
- test_time.py: 30 TN (test_norm) + 19 ITN (test_denorm) hand-written cases
- Full ar suite: 355 passed
Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>…. tests 399 passed. Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
…months) Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
Signed-off-by: Enas Albasiri <ealbasiri@gradcenter.cuny.edu>
folivoramanh
left a comment
There was a problem hiding this comment.
- remove unuse import
- make sure pytest and sparrowhawk test pass
- update jenkins ar cache date
There was a problem hiding this comment.
FAILED tests/nemo_text_processing/ar/test_time.py::TestTime::test_denorm_07__ - AssertionError: input: الثانية عشرة
assert '0' == '12'
- 12
- 0
make sure pytest and sparrowhawk test pass
| ستة وثمانية آلاف وتسع مئة وأربعة وأربعين من عشرة آلاف في المائة~%6,8944 | ||
| ستة آلاف وسبع مئة وثمانية وتسعين وتسعة من عشرة في المائة~%6798,9 | ||
| ست مئة وثمانية وخمسين وثمان مئة وثمانية وثمانين من ألف في المائة~%658,888 | ||
| اثنين وثمانية من عشرة في المائة~2.8% |
There was a problem hiding this comment.
this case doensn't have %, why it can normalize and have %?
| @@ -2,13 +2,13 @@ | |||
| تسعة فهرنهايت~f9 | |||
| اثنا عشر في المائة~12% | |||
| تسعة في المائة~09% | |||
There was a problem hiding this comment.
if the input is only 09, how to know if it is cardinal or measure?
There was a problem hiding this comment.
data money got more values than $ €, can you add more test for other money measure?
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
What does this PR do ?
Adds the Arabic (
ar) grammar contributions: expanded cardinals, time, and date — each in both text normalization (TN) and inverse text normalization (ITN).Scope (3 semiotic areas)
Cardinal (TN + ITN)
Time (TN + ITN)
Date (TN + ITN)
-,/,\,.(dotted requires full d.m.y so decimals stay decimals) and spelled-out months.dd/mm[/yyyy](or month name + year), Hijri suffix preserved.Testing
pytest --cpufullarsuite: 399 passed, 0 failed.Before your PR is "Ready for review"
Pre checks:
git commit -s)pytest --cpu— 399 passed. (Sparrowhawk: not run.)pytesttest cases (cardinal/time/date + regenerated dependent data). (Sparrowhawk cases: not added.)__init__.pyfor new data folders (months, ordinal, time).en/graph_utils.pycopy header — N/A.aralready documented).tools/text_processing_deployment/pynini_export.py—aralready registered.PR Type: