DeepEntityMatching

Entity resolution in PyTorch
Train a classifier to find entity matching between two sources.

Slides

https://rs9000.github.io/assets/docs/slide.pdf

Word embedding

This model require GloVe or another word embedding
https://nlp.stanford.edu/projects/glove/

How to use

Train classifier

usage: train.py [-args]
arguments:
--source1 Source file 1
--source2 Source file 2
--separator Char separator in CSV source files
--n_attrs Number of attributes in sources files --mapping Partial ground truth mapping of sources files
--blocking_size Window size of the blocking method (Sorted Neighbourhood)
--blocking_attr Attributes of blocking
--word_embed Word embedding file
--word_embed_size Word embedding vector size
--save_model Save trained model
--load_model Load pre-trained model

Merge tables with pre-trained classifier

usage: merge_csv.py [-args]
arguments:
--source1 Source file 1
--source2 Source file 2
--output_file Output file
--separator Char separator in CSV source files
--n_attrs Number of attributes in sources files --blocking_size Window size of the blocking method (Sorted Neighbourhood)
--blocking_attr Attributes of blocking
--word_embed Word embedding file
--word_embed_size Word embedding vector size
--load_model Load pre-trained model

Output

Train

Loading Glove Model...
...Done! 400001 words loaded!
NLP(
(fc1): Linear(in_features=5, out_features=50, bias=True)
(fc2): Linear(in_features=50, out_features=2, bias=True)
(probs): LogSoftmax()
)
Start training...
Epoch: 0
Tot Loss: 3.6684898769376177
Accuracy: 0.6945840312674484
#True Positive: 8 #FP: 138
#True Negative: 1236 #FN 409
....
Epoch: 10
Tot Loss: 0.20634120076961332
Accuracy: 0.954215522054718
#True Positive: 381 #FP: 46
#True Negative: 1328 #FN 36

Merge

Loading Glove Model...
...Done! 400001 words loaded!
Start Merging... (it may take a while)
Found: 1580 duplicates
Merging files....
Done!
File created: clean_table.csv