Port transductive alignment model

[pmachapman]

Fixes#455 and #473.

Port of sillsdev/machine.py#327

---

This change is 

[ddaspit]

@ddaspit reviewed 9 files and all commit messages, and made 3 comments.
Reviewable status: 9 of 10 files reviewed, 3 unresolved discussions (waiting on Enkidu93 and pmachapman).

---

src/SIL.Machine/Corpora/CorporaExtensions.cs line 1135 at r1 (raw file):

 }
public static IParallelTextCorpusWordAlign(

I think there are too many pitfalls with trying to implement this operation. Instead, we should add a similar IParallelTextCorpus extension method in SIL.Machine.Translation.Thot that trains and aligns a corpus using a Thot model. The method should accept a ThotWordAlignmentModelType parameter. You can have overloads that accept a ThotWordAlignmentModel or ThotSymmetrizedWordAlignmentModel as well. It would mimic the word_align_corpus function in Machine.py. If the passed in model is not configured to emit training alignments, then it should fall back to inductive alignment.

---

src/SIL.Machine.Translation.Thot/ThotWordAlignmentModel.cs line 169 at r1 (raw file):

 {
CheckDisposed();
IntPtr nativeMatrix =Thot.AllocNativeMatrix(_sourceWords.Count, _targetWords.Count);

To get the size of the matrix, you can call swAlignModel_getTrainingAlignment with a null buffer.

---

src/SIL.Machine.Translation.Thot/ThotWordAlignmentModelTrainer.cs line 24 at r1 (raw file):

 private readonly bool _isEflomal;
public ThotWordAlignmentModelTrainer(

You forgot emitTrainingAlignments here.

[pmachapman]

@pmachapman made 3 comments.
Reviewable status: 9 of 10 files reviewed, 3 unresolved discussions (waiting on ddaspit and Enkidu93).

---

src/SIL.Machine/Corpora/CorporaExtensions.cs line 1135 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

I think there are too many pitfalls with trying to implement this operation. Instead, we should add a similar IParallelTextCorpus extension method in SIL.Machine.Translation.Thot that trains and aligns a corpus using a Thot model. The method should accept a ThotWordAlignmentModelType parameter. You can have overloads that accept a ThotWordAlignmentModel or ThotSymmetrizedWordAlignmentModel as well. It would mimic the word_align_corpus function in Machine.py. If the passed in model is not configured to emit training alignments, then it should fall back to inductive alignment.

I think I understand what you mean, but still have a couple of questions (and think I might have misunderstood some of your instructions) - see the updated code committed below. Can we talk about this in the meeting tomorrow?

---

src/SIL.Machine.Translation.Thot/ThotWordAlignmentModel.cs line 169 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

To get the size of the matrix, you can call swAlignModel_getTrainingAlignment with a null buffer.

Unless I misunderstand what is going on, passing IntPtr.Zero to the matrix argument crashes with an access violation as the matrix is clamped:

doubleswAlignModel_getTrainingAlignment(void* swAlignModelHandle, unsignedint n, bool** matrix, unsignedint* iLen,
unsignedint* jLen)
{
auto alignmentModel = static_cast<AlignmentModel*>(swAlignModelHandle);
WordAlignmentMatrix waMatrix;
LgProb prob = alignmentModel->getTrainingAlignment(n, waMatrix);
// A filtered/out-of-range pair yields an empty matrix, so clamp to its actual// dimensions as well as the caller-provided capacity.for (unsignedint i = 0; i < *iLen && i < waMatrix.get_I(); i++)
for (unsignedint j = 0; j < *jLen && j < waMatrix.get_J(); j++)
matrix[i][j] = waMatrix.getValue(i, j);
*iLen = waMatrix.get_I();
*jLen = waMatrix.get_J();
return prob;
}

If it sounds good to you, I can create a PR to remove the clamping in thot? I think this will be OK as we output the X and Y lengths.

---

src/SIL.Machine.Translation.Thot/ThotWordAlignmentModelTrainer.cs line 24 at r1 (raw file):

Previously, ddaspit (Damien Daspit) wrote…

You forgot emitTrainingAlignments here.

Done. Thanks!

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.17886% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.82%. Comparing base (37e13b7) to head (8110674).

Files with missing lines	Patch %	Lines
....Translation.Thot/IParallelTextCorpusExtensions.cs	87.93%	5 Missing and 2 partials ⚠️
...nslation.Thot/ThotSymmetrizedWordAlignmentModel.cs	84.37%	3 Missing and 2 partials ⚠️
...Machine/Corpora/WordAlignParallelTextCorpusBase.cs	57.14%	3 Missing ⚠️
....Translation.Thot/ThotWordAlignmentModelTrainer.cs	88.88%	1 Missing ⚠️
src/SIL.Machine/Corpora/CorporaExtensions.cs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## master #466 +/- ##
==========================================
+ Coverage 73.51% 73.82% +0.31% 
==========================================
Files 451 454 +3 Lines 37692 37809 +117 Branches 5183 5193 +10 ==========================================
+ Hits 27708 27914 +206 + Misses 8845 8748 -97 - Partials 1139 1147 +8 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Enkidu93]

@Enkidu93 reviewed 13 files and all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on ddaspit).