A Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks
Victor SANH, Thomas WOLF, Sebastian RUDER
Accepted at AAAI 2019
HMTL is a Hierarchical Multi-Task Learning model which combines a set of four carefully selected semantic tasks (namely Named Entity Recoginition, Entity Mention Detection, Relation Extraction and Coreference Resolution). The model achieves state-of-the-art results on Named Entity Recognition, Entity Mention Detection and Relation Extraction. Using SentEval, we show that as we move from the bottom to the top layers of the model, the model tend to learn more complex semantic representation.
For further details on the results, we refer to our paper.
We release the code for training, fine tuning and evaluating HMTL. We hope that this code will be useful for building your own Multi-Task models (hierarchical or not). The code is written in Python and powered by Pytorch.
The main dependencies are:
The code works with Python 3.6. A stable version of the dependencies is listed in requirements.txt.
You can quickly setup a working environment by calling the script ./script/machine_setup.sh. It installs Python 3.6, create a clean virtual environment, and install all the required dependencies (listed in requirements.txt). Please adapt the script depending on your needs.
We base our implementation on the AllenNLP library. For an introduction to this library, you should check these tutorials.
An experiment is defined in a json configuration file (see configs/*.json for examples). The configuration file mainly describes the datasets to load, the model to create along with all the hyper-parameters of the model.
Once you have set up your configuration file (and defined custom classes such DatasetReaders if needed), you can simply launch a training with the following command and arguments:
python train.py --config_file_path configs/hmtl_coref_conll.json --serialization_dir my_first_trainingOnce the training has started, you can simply follow the training in the terminal or open a Tensorboard (please make sure you have installed Tensorboard and its Tensorflow dependecy before):
tensorboard --logdir my_first_training/logWe used SentEval to assess the linguistic properties learned by the model. hmtl_senteval.py gives an example of how we can create an interface between SentEval and HMTL. It evaluates the linguistic properties learned by every layer of the hiearchy (shared based word embeddings and encoders).
To download the pre-trained embeddings we used in HMTL, you can simply launch the script ./script/data_setup.sh.
We do not attach the datasets used to train HMTL for licensing reasons, but we invite you to collect them by yourself: OntoNotes 5.0, CoNLL2003, and ACE2005. The configuration files expect the datasets to be placed in the data/ folder.
Please consider citing the following paper if you find this repository useful.
@article{sanh2018hmtl,
title={A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks},
author={Sanh, Victor and Wolf, Thomas and Ruder, Sebastian},
journal={arXiv preprint arXiv:1811.06031},
year={2018}
}