Re-enable rspec of failing tests

[ronaldtse]

For #23.

[ronaldtse]

To run the tests: run bundle exec rake and the test suite will be run.

[ronaldtse]

UPDATED failures:

1) Rababa::Diacritizer diacriticizes قطر
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "قَطَر"
got: "قَطُر"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'
2) Rababa::Diacritizer diacriticizes وقال ادخلوا مصر إن شاء الله آمنين
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "وَقَالَ ادْخُلُوا مِصْرَ إِن شَاءَ اللَّهُ آمِنِينَ"
got: "وَقَالَ اُدْخُلُوا مِصْرَ إِنْ شَاءَ اللَّهُ آمِنِيْن"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'
3) Rababa::Diacritizer diacriticizes يذهب المسلمون كل عام إلى المملكة العربية السعودية لأداء مناسك الحج
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "يَذْهِبُ الْمُسْلِمُونَ كُلَّ عَامٍ إِلَى الْمَمْلَكَةِ الْعَرَبِيَّةِ السُّعُودِيَّةِ لِأَدَاءِ مَنَاسِكِ الْحَجِّ"
got: "يُذْهَبُ المُسْلِمُونَ كُلَّ عَامٍ إِلَى المَمْلَكَةِ العَرَبِيَّةِ السُّعُودِيَّةِ لِأَدَاءِ مَنَاسِكِ الحَج"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'
4) Rababa::Diacritizer diacriticizes لقد كان في يوسف وإخوته آيات للسائلين
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "لَقَدْ كَانَ فِي يُوسُفَ وَإِخْوَتِهِ آيَاتٌ لِلسَّائِلِينَ"
got: "لَقَدْ كَانَ فِي يُوسُفَ وَإِخْوَتِهِ آيَاتٌ لِلسَّائِلِين"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'
5) Rababa::Diacritizer diacriticizes الحمد لله رب العالمين
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "الْحَمْدُ لِلَّهِ رَبِّ الْعَالَمِينَ"
got: "الحَمْدُ لِلَّهِ رَبِّ العَالَمِين"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'
6) Rababa::Diacritizer diacriticizes وما كان الله ليعذبهم وأنت فيهم
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "وَمَا كَانَ اللَّهُ لِيُعَذِّبَهُمْ وَأَنْتَ فِيهِمْ"
got: "وَمَا كَانَ اللَّهُ لِيُعَذِّبَهُمْ وَأَنْتَ فِيهِم"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'
7) Rababa::Diacritizer diacriticizes نحن نقص عليك أحسن القصص
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "نَحْنُ نَقُصُّ عَلَيْكَ أَحْسَنَ الْقَصَصِ"
got: "نَحْنُ نَقُصّ عَلَيْكَ أَحْسَنَ القَصَص"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'
8) Rababa::Diacritizer diacriticizes سأذهب إلى برج eiffel
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "سَأُذْهِبُ إِلَى بُرْجِ eiffel"
got: "سَأَذْهَبُ إِلَى بُرْج eiffel"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'
9) Rababa::Diacritizer diacriticizes # گيله پسمير الجديد 34
Failure/Error: expect(diacritizer.diacritize_text(source)).to eq target
expected: "# گيَلِهُ پسُمِيْرٌ الجَدِيدُ 34"
got: "# گيَلِهُ پسُمِيْرٌ الجَدِيد 34"
(compared using ==)
# ./spec/rababa/diacritizer_spec.rb:82:in `block (3 levels) in <top (required)>'

[ronaldtse]

One transformation that seems to fail many examples is this one:

1. It converts "اللَّهُ" to "الللَّهُ": there is one character inserted in between?

[ronaldtse]

2. The other common failure is the missing diacritic at the last character of a sentence.

[ronaldtse]

One transformation that seems to fail many examples is this one:

1. It converts "اللَّهُ" to "الللَّهُ": there is one character inserted in between?

Confirm that this issue is fixed in 312ff4a.

[ronaldtse]

@AhMohsen46 Upon looking closer there are some minor failures, can you comment on them individually? Do you think more training by @gilgameshjw can solve these issues?

• Qatar:
  expected: "قَطَر"
  got: "قَطُر"

• missing last diacritic: (7 tests)
  expected: "وَقَالَ ادْخُلُوا مِصْرَ إِن شَاءَ اللَّهُ آمِنِينَ"
  got: "وَقَالَ اُدْخُلُوا مِصْرَ إِنْ شَاءَ اللَّهُ آمِنِيْن"

• missing diacritic in the middle of sentence (last char of word) (1 test)
  expected: "نَحْنُ نَقُصُّ عَلَيْكَ أَحْسَنَ الْقَصَصِ"
  got: "نَحْنُ نَقُصّ عَلَيْكَ أَحْسَنَ القَصَص"

[gilgameshjw]

@AhMohsen46 Upon looking closer there are some minor failures, can you comment on them individually? Do you think more training by @gilgameshjw can solve these issues?

* Qatar:
expected: "قَطَر"
got: "قَطُر"
* missing last diacritic: (7 tests)
expected: "وَقَالَ ادْخُلُوا مِصْرَ إِن شَاءَ اللَّهُ آمِنِينَ"
got: "وَقَالَ اُدْخُلُوا مِصْرَ إِنْ شَاءَ اللَّهُ آمِنِيْن"

I have proposed as solution to create words dictionaries with our geodata and the data from the training dataset.

We could then match diacritized text to the closest real word candidate and thus increase accuracy, IMO.