Use new machine eflomal implementation by Enkidu93 · Pull Request #336 · sillsdev/machine.py

Enkidu93 · 2026-07-23T15:34:31Z

Also:

Train on training parallel data as well as pretranslation data when running alignments during an MT job
Fix a few copy-paste errors floating in word alignment build job classes

This update is a little difficult to test. I have tested it E2E and the Serval E2E tests do pass. The scores are populated properly. I compared the alignments with the previous version for the E2E test data and they seem mostly similar - maybe the old ones are a little better 😬 - but it's difficult to judge. I'm going to trust that, as long as I am using the API properly, switching to the new implementation and training the aligner on the training data is an improvement. My outstanding questions are:

Am I using the alignment API correctly? Am I preparing the training and inference data correctly? I followed what was in the word alignment build job but I honestly don't have a lot of confidence in that since I noticed a couple things that seem to be bugs/typos. I do wonder as well about the call to lowercase() - see my TODO comments.
Is this the right way to configure things? Or do we want a separate NMT word alignment configuration section?

This change is

codecov-commenter · 2026-07-23T15:48:06Z

Codecov Report

❌ Patch coverage is 93.97590% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.10%. Comparing base (8dedd19) to head (cda9091).

Files with missing lines	Patch %	Lines
machine/jobs/word_alignment_file_service.py	33.33%	6 Missing ⚠️
machine/jobs/nmt_engine_build_job.py	94.93%	4 Missing ⚠️
machine/jobs/word_alignment_build_job.py	95.45%	2 Missing ⚠️
machine/corpora/corpora_utils.py	0.00%	1 Missing ⚠️
machine/corpora/flatten.py	85.71%	1 Missing ⚠️
...ine/jobs/thot/thot_word_alignment_model_factory.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## main #336 +/- ##
==========================================
+ Coverage 92.02% 92.10% +0.08% 
==========================================
Files 389 389 Lines 24485 24559 +74 ==========================================
+ Hits 22532 22620 +88 + Misses 1953 1939 -14

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ddaspit

@ddaspit made 2 comments.
Reviewable status: 0 of 10 files reviewed, 2 unresolved discussions (waiting on Enkidu93).

machine/jobs/nmt_engine_build_job.py line 182 at r1 (raw file):

parallel_training_corpus.get_rows(),
)
) # TODO .lowercase()?

We should perform Unicode and casing normalization.

machine/jobs/nmt_engine_build_job.py line 196 at r1 (raw file):

parallel_pretranslation_rows=list(parallel_pretranslation_corpus) # TODO .lowercase()?alignments=alignment_model.align_batch(parallel_pretranslation_rows)

This will work, but it would be better to use the transductive API, i.e. get_training_alignment. align_batch will be more expensive.

Enkidu93

@Enkidu93 made 2 comments.
Reviewable status: 0 of 10 files reviewed, 2 unresolved discussions (waiting on ddaspit).

machine/jobs/nmt_engine_build_job.py line 182 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

We should perform Unicode and casing normalization.

OK, thank you 👍, but when we actually go to populate the source and target tokens in the pretranslation data structure, we will want them to be exactly as they were in the original text, right? Otherwise, the marker placement won't work properly. Or should we update the code in the marker placement handler to only look for normalized substrings?

machine/jobs/nmt_engine_build_job.py line 196 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

This will work, but it would be better to use the transductive API, i.e. get_training_alignment. align_batch will be more expensive.

Would you like me only allow transductive models or just use the transductive API if the model supports it and otherwise default to this?

ddaspit

@ddaspit reviewed 14 files and all commit messages, made 3 comments, and resolved 2 discussions.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on Enkidu93).

machine/jobs/nmt_engine_build_job.py line 182 at r1 (raw file):

Previously, Enkidu93 (Eli C. Lowry) wrote…

OK, thank you 👍, but when we actually go to populate the source and target tokens in the pretranslation data structure, we will want them to be exactly as they were in the original text, right? Otherwise, the marker placement won't work properly. Or should we update the code in the marker placement handler to only look for normalized substrings?

You are correct. I added a commit that inferences on the normalized tokens but returns the unnormalized tokens. It also added transductive API support, a full streaming pipeline, and fixed a bug in flatten.

machine/jobs/nmt_engine_build_job.py line 196 at r1 (raw file):

Previously, Enkidu93 (Eli C. Lowry) wrote…

Would you like me only allow transductive models or just use the transductive API if the model supports it and otherwise default to this?

I added support for the transductive API.

machine/jobs/word_alignment_build_job.py line 86 at r2 (raw file):

returntrain_corpus_sizedef_batch_inference(

This should be updated to work similar to NmtEngineBuildJob._align.

Enkidu93

@Enkidu93 partially reviewed 1 file and made 3 comments.
Reviewable status: 11 of 14 files reviewed, 1 unresolved discussion (waiting on ddaspit).

machine/jobs/nmt_engine_build_job.py line 182 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

You are correct. I added a commit that inferences on the normalized tokens but returns the unnormalized tokens. It also added transductive API support, a full streaming pipeline, and fixed a bug in flatten.

OK, great 👍

machine/jobs/nmt_engine_build_job.py line 196 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

I added support for the transductive API.

Great, thank you

machine/jobs/word_alignment_build_job.py line 86 at r2 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

This should be updated to work similar to NmtEngineBuildJob._align.

Done. Of course, the word alignment API isn't really set up yet to take advantage of the transductive model. For now, I've just had it use align_batch, but maybe we should make an issue to update how training/inferencing work for word alignment in general to take advantage of this. Even if we allow for training and inferencing on the same data (which we probably just should for word alignment in general?), we'd still need a way to index the training data so we can retrieve the alignments.

ddaspit

@ddaspit reviewed 4 files and all commit messages, made 2 comments, and resolved 1 discussion.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on Enkidu93).

machine/jobs/word_alignment_build_job.py line 97 at r4 (raw file):

 ) ->None:
inference_inputs=self._word_alignment_file_service.get_word_alignment_inputs()

In order to make it worthwhile to spool the data to disk, get_word_alignment_inputs should return a generator instead of a list.

machine/jobs/word_alignment_build_job.py line 135 at r4 (raw file):

check_canceled()
segments= [(lowercase(row.source_segment), lowercase(row.target_segment)) for_, rowinwa_batch]
alignments=alignment_model.align_batch(segments)

Progress is never advanced while inferencing.

Enkidu93

@Enkidu93 reviewed 14 files and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on ddaspit).

machine/jobs/word_alignment_build_job.py line 97 at r4 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

In order to make it worthwhile to spool the data to disk, get_word_alignment_inputs should return a generator instead of a list.

Done.

machine/jobs/word_alignment_build_job.py line 135 at r4 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

Progress is never advanced while inferencing.

Done.

ddaspit

@ddaspit reviewed 3 files and all commit messages, made 1 comment, and resolved 2 discussions.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on Enkidu93).

Enkidu93 · 2026-08-10T20:28:27Z

Decided to test this E2E - and I'm glad I did because I hit an error that I wasn't hitting before:

Traceback (most recent call last):
File "/root/.clearml/venvs-builds/code/untitled.py", line 13, in <module>
run(args)
File "/usr/local/lib/python3.12/site-packages/machine/jobs/build_nmt_engine.py", line 74, in run
raise e
File "/usr/local/lib/python3.12/site-packages/machine/jobs/build_nmt_engine.py", line 64, in run
train_corpus_size, _ = job.run(progress, check_canceled)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/machine/jobs/translation_engine_build_job.py", line 45, in run
self._batch_inference(parallel_corpus, progress_reporter, check_canceled)
File "/usr/local/lib/python3.12/site-packages/machine/jobs/nmt_engine_build_job.py", line 134, in _batch_inference
for pt_info in results:
^^^^^^^
File "/usr/local/lib/python3.12/site-packages/machine/jobs/nmt_engine_build_job.py", line 231, in _align
transductive_model.get_training_alignment(index + i) for i in range(len(pt_batch))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/machine/translation/thot/thot_symmetrized_word_alignment_model.py", line 74, in get_training_alignment
_, matrix = self._aligner.get_training_alignment(n)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MemoryError: std::bad_array_new_length

Not my top priority right now but just so you know why I haven't merged this yet.

Enkidu93 · 2026-08-12T19:31:04Z

This looks to be caused by having empty pretranslations. Peter is going to take a look on the thot side since he's already making updates to that code.

Enkidu93

Alright, the slow-down was from progress-reporting. I've reduced the reporting significantly and now the whole alignment phase takes a few minutes rather than twenty. We could reduce it further or eliminate it altogether if you prefer. I've also updated thot to the new version that fixes the error above. @ddaspit, I think this is ready for re-review.

@Enkidu93 made 1 comment.
Reviewable status: 14 of 18 files reviewed, all discussions resolved (waiting on ddaspit).

ddaspit

Good job tracking this down. What kind of an impact on speed does it have if we remove progress reporting during alignment inferencing altogether?

@ddaspit reviewed 4 files and all commit messages, and made 2 comments.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on Enkidu93).

machine/translation/thot/thot_word_alignment_model_trainer.py line 195 at r12 (raw file):

 )
cur_step=0update_frequency=num_steps//10ifnum_stepsandnum_steps>10elseNone

I remember us running into a similar problem a while ago. We do progress throttling in the get_clearml_progress_caller function. Was that not doing a good enough job? Should that function be updated instead of this? At the very least, we should add a comment explaining the issue.

Enkidu93

It's about comparable if we remove it altogether actually - I just confirmed. It's still taking 2-5 minutes.

@Enkidu93 made 2 comments.
Reviewable status: 17 of 18 files reviewed, 1 unresolved discussion (waiting on ddaspit).

machine/translation/thot/thot_word_alignment_model_trainer.py line 195 at r12 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

I remember us running into a similar problem a while ago. We do progress throttling in the get_clearml_progress_caller function. Was that not doing a good enough job? Should that function be updated instead of this? At the very least, we should add a comment explaining the issue.

Yes, the throttling there doesn't seem to do much. The iteration reporting in ClearML is very slow. We are already throttling NMT training in the trainer similarly: #227. If you'd prefer, we could try to have a uniform way of throttling them in the cleaml progress function. We could also pass the update_frequency to the constructor if you prefer. Part of the reason unifying them and throttling them in one place might be difficult is that you may genuinely want the frequency to differ task to task.

ddaspit

@ddaspit reviewed 1 file and all commit messages, made 2 comments, and resolved 1 discussion.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on Enkidu93).

machine/translation/thot/thot_word_alignment_model_trainer.py line 195 at r12 (raw file):

Previously, Enkidu93 (Eli C. Lowry) wrote…

Yes, the throttling there doesn't seem to do much. The iteration reporting in ClearML is very slow. We are already throttling NMT training in the trainer similarly: #227. If you'd prefer, we could try to have a uniform way of throttling them in the cleaml progress function. We could also pass the update_frequency to the constructor if you prefer. Part of the reason unifying them and throttling them in one place might be difficult is that you may genuinely want the frequency to differ task to task.

I'm good with tailoring the throttling for each task.

Remove redundant alignment score Fix api usage; adjust test Remove unused import Properly mock word alignment; revert settings Fix logging; do not include NULL alignments Make naming more consistent Revert settings.yml change Use transductive API for getting alignments - full streaming pipeline for inferencing - fall back to inductive API - fix text id handling in "flatten" Update word alignment build job to properly use alignment API and batch Address reviewer comments Update word alignment job test mock to return correct type Add logging for debugging Revert debugging changes Debug commit Update thot; report steps (debug) Debug steps and reporting Remove debug logging; add proper align phase Debug logging Fix update frequency setting Remove debug logging; make update frequency dynamic Remove reporting (debug commit) Fix typo (debug) Remove debug changes; add comment Fix throttling logic typo

Enkidu93 requested a review from ddaspit July 23, 2026 15:34

Enkidu93 mentioned this pull request Jul 23, 2026
Calculate basic alignment score for word pairs from the sentence_score #316
Closed

ddaspit requested changes Jul 23, 2026
View reviewed changes

Enkidu93 commented Jul 23, 2026
View reviewed changes

ddaspit reviewed Jul 24, 2026
View reviewed changes

Enkidu93 force-pushed the use_new_eflomal_implementation branch from 68cff34 to 9cdec1aCompare July 28, 2026 17:08

Enkidu93 commented Jul 29, 2026
View reviewed changes

Enkidu93 requested a review from ddaspit August 5, 2026 14:01

ddaspit reviewed Aug 6, 2026
View reviewed changes

Enkidu93 commented Aug 10, 2026
View reviewed changes

ddaspit approved these changes Aug 10, 2026
View reviewed changes

pmachapman mentioned this pull request Aug 13, 2026
Fix crash when initializing a WordAlignmentMatrix with one dimension sillsdev/thot#16
Merged

Enkidu93 force-pushed the use_new_eflomal_implementation branch from 98b9e6f to ab22a84Compare August 20, 2026 19:14

Enkidu93 commented Aug 20, 2026
View reviewed changes

ddaspit requested changes Aug 20, 2026
View reviewed changes

Enkidu93 commented Aug 20, 2026
View reviewed changes

ddaspit approved these changes Aug 20, 2026
View reviewed changes

Enkidu93 force-pushed the use_new_eflomal_implementation branch from eaf93da to cda9091Compare August 26, 2026 19:57

Enkidu93 merged commit cb1eff9 into mainAug 26, 2026
16 of 17 checks passed

Enkidu93 deleted the use_new_eflomal_implementation branch August 26, 2026 20:47

Enkidu93 mentioned this pull request Aug 26, 2026
Port 'Use new machine eflomal implementation' sillsdev/machine#487
Open

Uh oh!

Uh oh!

Conversation

Enkidu93 commented Jul 23, 2026• edited by ddaspit Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jul 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 commented Aug 10, 2026

Uh oh!

Enkidu93 commented Aug 12, 2026

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Enkidu93 left a comment

Choose a reason for hiding this comment

Uh oh!

ddaspit left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Enkidu93 commented Jul 23, 2026•
edited by ddaspit
Loading

codecov-commenter commented Jul 23, 2026•
edited
Loading