This work started as an attempt to reproduce Yu et al.'s http://arxiv.org/abs/1412.1632
Used word embeddings: pre-trained GloVe vectors from http://nlp.stanford.edu/projects/glove/
So far implemented:
- Bag of words + basic gradient descent learning classification
- Bag of words + basic gradient descent learning classification + word counts logistic regression
For sentence selection development, used dataset: TREC-based originally by Wang et al., 2007; in the form by Yao et al., 2013 as downloaded from https://code.google.com/p/jacana/
Preprocessing (not required):
- Run save.py first with updated filepath constants (const.py) if you have different dataset (requires jacana formating)
Train and test:
Run train.py for training from TREC TRAIN dataset and testing from TREC TEST dataset
train.py generates truth.txt and res.txt, to evaluate using the official trec_eval tool, run
trec_eval -a truth.txt res.txt
TODO:
- CNN instead of bag of words unigram averaging for aggregate embeddings.
Results (evaluated using stock TREC scripts):
| MRR | MAP | |
|---|---|---|
| TRAIN | 0.7312 | 0.6551 |
| TRAIN-ALL | 0.7308 | 0.6566 |
| TRAIN+count | 0.7763 | 0.7165 |
| TRAIN-ALL+count | 0.8128 | 0.7258 |
Folow these steps if you want to retrain currently used weights:
Gather input data (labelled tuples) according to the instructions in YodaQA data/ml/embsel/README.md.
Run './std_run.sh -p PATH' (PATH is the directory of dumped yodaqa files). You can alter the training constants in basicgrad.py and train.py.
If you are happy with the results, you copy the generated file data/Mbtemp.txt to yodaqa src/main/resources/cz/brmlab/yodaqa/analysis/rdf/Mbprop.txt
In summary, use this:
./std_run.sh -p ../yodaqa/data/ml/embsel/propdata
cp data/Mbtemp.txt ../yodaqa/src/main/resources/cz/brmlab/yodaqa/analysis/rdf/Mbprop.txt
(With a random 1:1 train:test split of the original curated-train.)
Used dataset:
train questions: 270 train sentences: 19624 (generated with curated-measure.sh train)
test questions: 222 test sentences: 17561 (generated with curated-measure.sh train)
2.7902739024% of the properties contains correct answers
random test mrr = 0.0475542678953
Current results:
MMR after unigram learning train: 0.600856454434
MMR after unigram learning test: 0.582881935037
Folow these steps if you want to retrain currently used weights:
Gather input data (labelled tuples) according to the instructions in YodaQA data/ml/embsel/README.md.
Run './std_run.sh -p PATH' (PATH is the directory of dumped yodaqa files). You can alter the training constants in basicgrad.py and train.py.
If you are happy with the results, you copy the generated file data/Mbtemp.txt to yodaqa src/main/resources/cz/brmlab/yodaqa/analysis/passextract/Mb.txt
In summary, use this (with YodaQA's f/sentence-selection branch):
./std_run.sh ../yodaqa/data/ml/embsel/sentdata
cp data/Mbtemp.txt ../yodaqa/src/main/resources/cz/brmlab/yodaqa/analysis/passextract/Mb.txt
(With a random 1:1 train:test split of the original curated-train.)
Used dataset:
train questions: 186 train sentences: 43843 (generated with curated-measure.sh train)
test questions: 429 test sentences: 88779 (generated with curated-measure.sh test)
5.21294450264% of the properties contains correct answers
random test mrr = 0.0760195275186
Current results:
baseline (clue1+0.25*clue2):
MRR unigram+clues train 0.249327071552
MRR unigram+clues test 0.29659580682
glove only:
MMR after unigram learning train: 0.224787152966
MMR after unigram learning test: 0.222749753007
glove+clue1:
MRR unigram+clues train 0.358206351223
MRR unigram+clues test 0.388948882077