he_treebanks

A library for handling Hebrew treebanks, with focus on the Hebrew UD treebank HTB.

Structure

Data

1. The UD_HTB directory contains the Hebrew UD treebank, divided into dev, train and test splits.
2. Conversion schemes - mostly json files that hold the conversion information to be used by the conversion scripts.

src

mostly conversion scripts. The main ones are:

changes_to_treebank.py

A tool for investigation and conversion of the treebank. e.g. allows retrieval of all tokens that correspond to some attribute. For previous changes, see changelog.

lattices_builder.py

takes a raw sentence and builds a "lattices" file - one which all possible parses of each token are listed. For example, the sentence שלום, היום יום יפה would yield the following lattice (in a conll-ul format):

FROM TO FORM LEMMA UPOS CPOS FEATURES MISC ANCHORS
0-5 בצלם
0 5 BCLM BCLM PROPN -
0 1 B B ADP - goldid=1
0 3 BCL BCL NOUN - Gender=Masc|Number=Sing
1 2 H H DET - PronType=Art goldid=2
1 3 CL CL NOUN - Gender=Masc|Number=Sing
2 3 CL CL NOUN - Gender=Masc|Number=Sing goldid=3
2 5 CLM CILM VERB - Gender=Masc|Number=Si...
2 5 CLM CILM VERB - Gender=Masc|Mood=Imp...
2 5 CLM CLM NOUN - Definite=Cons|Gender=Mas...
2 5 CLM CLM NOUN - Gender=Masc|Number=Sing
3 4 FL FL ADP - goldid=4
4 5 HM ANI PRON - Gender=Masc|Number=Plur|Person=3 goldid=5
5-7 הנעים
5 7 HNEIM HNEIM VERB - Gender=Masc...
5 6 H H DET - PronType=Art goldid=6
5 6 H H SCONJ -
6 7 NEIM NEIM ADJ - Gender=Masc|Number=Sing goldid=7
6 7 NEIM NEIM ADJ - Definite=Cons|Gender=...
6 7 NEIM NEIM ADV - Polarity=Neg
6 7 NEIM NEIM AUX - Gender=Masc|Number=Sing...
6 7 NEIM NEIM PROPN -
6 7 NEIM NEIM VERB - Gender=Masc|Tense=Part|VerbForm=Part
6 7 NEIM NEIM VERB - Gender=Masc|VerbForm=Part

Notebooks

Jupyter notebook versions of the scripts found in src.