Description

reference pytorch code for named entity tagging.

embedding
- word : GloVe, BERT, DistilBERT, feature-based BERT using DSA(Dynamic Self Attention) pooling, SpanBERT, ALBERT, RoBERTa, BART, ELECTRA, ELMo
- character : CNN
- pos : lookup
encoding
- BiLSTM
- DenseNet
  - Dynamic Self-Attention: Computing Attention over Words Dynamically for Sentence Embedding
  - a slightly modified DenseNet for longer dependency.
decoding
- Softmax, CRF
related: reference pytorch code for intent(sentence) classification

Requirements

python >= 3.6
pip install -r requirements.txt

pretrained embedding

glove

download GloVe6B and unzip to 'embeddings' dir

$ mkdir embeddings
$ ls embeddings
glove.6B.zip
$ unzip glove.6B.zip

BERT, ALBERT, RoBERTa, BART, ELECTRA(huggingface's transformers)
SpanBERT
- pretrained SpanBERT models are compatible with huggingface's BERT modele except 'bert.pooler.dense.weight', 'bert.pooler.dense.bias'.
ELMo(allennlp)

$ cd embeddings
$ curl -OL https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway_5.5B/elmo_2x4096_512_2048cnn_2xhighway_5.5B_weights.hdf5
$ curl -OL https://s3-us-west-2.amazonaws.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway_5.5B/elmo_2x4096_512_2048cnn_2xhighway_5.5B_options.json

Data

CoNLL 2003 (English)

from etagger

data/conll2003

data/conll2003_truecase

converting conll2003 data to its truecase

details

$ cd data/conll2003_truecase
$ python to-truecase.py --input_path ../conll2003/train.txt > train.txt
$ python to-truecase.py --input_path ../conll2003/valid.txt > valid.txt
$ python to-truecase.py --input_path ../conll2003/test.txt > test.txt

Naver NER 2019 (Korean)

from HanBert-NER

data/clova2019

converted to CoNLL data format.

details

이기범 eoj - B-PER
한두 eoj - O
쪽을 eoj - O
먹고 eoj - O
10분 eoj - B-TIM
후쯤 eoj - I-TIM
화제인을 eoj - B-CVL
먹는 eoj - O
것이 eoj - O
좋다고 eoj - O
한다 eoj - O
. eoj - O

data/clova2019_morph

tokenized by morphological analyzer and converted to CoNLL data format.

details

이기범 NNP - B-PER
한두 NNP - O
쪽 NNB - O
을 X-JKO - O
먹다 VV - O
고 X-EC - O
10 SN - B-TIM
분 X-NNB - I-TIM
후 NNG - I-TIM
쯤 X-XSN - I-TIM
화제 NNG - B-CVL
인 X-NNG - I-CVL
을 X-JKO - I-CVL
먹다 VV - O
...

'X-' prefix is prepending to POS(Part of Speech) tag of inside morphs for distinguishing following morphs.
we can evaluate the predicted result morph-by-morph or eojeol by eojeol manner(every lines having 'X-' POS tag are removed).

details

이기범 NNP - B-PER
한두 NNP - O
쪽 NNB - O
먹다 VV - O
10 SN - B-TIM
후 NNG - I-TIM
화제 NNG - B-CVL
먹다 VV - O
...

data/clova2019_morph_space

this data is identical to data/clova2019_morph except it treats spaces as tokens.

details

이기범 NNP - B-PER
_ _ - O
한두 NNP - O
_ _ - O
쪽 NNB - O
을 X-JKO - O
_ _ - O
먹다 VV - O
고 X-EC - O
_ _ - O
10 SN - B-TIM
분 X-NNB - I-TIM
_ _ - I-TIM
후 NNG - I-TIM
쯤 X-XSN - I-TIM
_ _ - O
화제 NNG - B-CVL
인 X-NNG - I-CVL
을 X-JKO - I-CVL
_ _ - O
먹다 VV - O
...

there is no test set. therefore, set valid.txt as test.txt.

KMOU NER 2019 (Korean)

from KMOU NER

data/kmou2019

build train.raw, valid.raw
- data version : https://github.com/kmounlp/NER/commit/0b32e066870bda9f65cc190f5e89c2edc6cf8f6d
- same as pytorch-bert-crf-ner
- train.raw : 00002_NER.txt, ..., EXOBRAIN_NE_CORPUS_007.txt (1,425 files)
- valid.raw : EXOBRAIN_NE_CORPUS_009.txt, EXOBRAIN_NE_CORPUS_010.txt (2 files)
- apply correction rules and converting to CoNLL data format

details

$ cd data/kmou2019
$ python correction.py -g train.raw > t
$ python to-conll.py -g t > train.txt
ex)
마음	마음	NNG	B-POH
’	’	SS	O
에	에	JKB	O
_	_	_	O
담긴	담기+ㄴ	VV+ETM	O
->
마음 NNG - B-POH
’ SS - O
에 JKB - O
_ _ - O
담기다 VV - O
ㄴ ETM - O
$ python correction.py -g valid.raw > t
$ python to-conll.py -g t > valid.txt

set valid.txt as test.txt

Korean pretrained model

Korean GloVe, BERT, ELECTRA, ELMo

description of Korean GloVe, BERT, DistilBERT, ELECTRA
- glove : kor.glove.300k.300d.txt (inhouse)
- bpe bert : kor-bert-base-bpe.v1, kor-bert-large-bpe (inhouse)
- dha-bpe bert : kor-bert-base-dha_bpe, kor-bert-large-dha_bpe (inhouse)
- dha bert : kor-bert-base-dha.v1, kor-bert-base-dha.v2 (inhouse)
- distil bpe bert : kor-distil-bpe-bert.v1 (inhouse)
- KoELECTRA-Base : koelectra-base-discriminator
- bpe electra : kor-electra-bpe-30k-512-1m (inhouse)
ELMo description
- kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_weights.hdf5, kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_options.json (inhouse)

CoNLL 2003 (English)

experiments summary

ntagger, measured by conlleval.pl (micro F1)

	F1 (%)	(truecase) F1 (%)	Features	GPU / CPU	CONDA	ONNX	Dynamic	Etc
GloVe, BiLSTM	88.23		word, pos	5.6217 / -	7.3838	3.6969		threads=14, packed
GloVe, BiLSTM	88.94		word, character, pos	6.4108 / -	9.5858	8.5656		threads=14, packed
GloVe, BiLSTM-CRF	90.14	90.26	word, character, pos	26.2807 / -	21.7474			threads=14, packed
GloVe, DenseNet-CRF	88.23		word, pos	24.7893 / -	22.5858			threads=14
GloVe, DenseNet-CRF	88.89		word, character, pos	28.0993 / -	25.2929			threads=14
BERT-tiny, BiLSTM	69.65		word	20.1376 / -
BERT-mini, BiLSTM	81.55		word	21.4632 / -
BERT-small, BiLSTM	86.35		word	22.6087 / -
BERT-medium, BiLSTM	88.29		word	27.0486 / -
DistilBERT, BiLSTM	89.50		word	13.4564 / -	58.9260	56.2819	47.8320	packed
BERT-base, BiLSTM-CRF	90.20		word	42.6464 / -				packed
BERT-base, BiLSTM	90.55		word	18.2323 / -	100.0505	79.1914	83.9590	packed
BERT-base, CRF	89.98		word	36.6893 / -
BERT-base	90.25		word	16.6877 / -	96.9004	72.8225	75.3025
BERT-base, BiLSTM	89.03		word	24.9076 / -				del 8,9,10,11, threads=14
BERT-large, BiLSTM	91.32	91.89	word	40.3581 / -				packed
BERT-large, BiLSTM	89.10		word	33.1376 / -				del 12 ~ 23
BERT-large, BiLSTM	86.11		word	49.3103 / -				BERT as feature-based, initial embedding
BERT-large, BiLSTM-CRF	86.43		word	63.1376 / -				BERT as feature-based, initial embedding
BERT-large, BiLSTM	89.72		word	47.9704 / -				BERT as feature-based, initial+first+last embedding
BERT-large, BiLSTM-CRF	89.96		word	67.2041 / -				BERT as feature-based, initial+first+last embedding
BERT-large, BiLSTM-CRF	89.67		word	68.7548 / -				BERT as feature-based, last embedding
BERT-large, BiLSTM-CRF	90.64		word	63.9397 / -				BERT as feature-based, [-4:] embedding
BERT-large, BiLSTM-CRF	90.52		word	70.8322 / -				BERT as feature-based, mean([0:3] + [-4:]) embedding
BERT-large, BiLSTM-CRF	90.81		word	68.6139 / -				BERT as feature-based, mean([0:17]) embedding
BERT-large, BiLSTM-CRF	90.76		word	60.8039 / -				BERT as feature-based, max([0:17]) embedding
BERT-large, BiLSTM-CRF	90.98		word	58.9112 / -				BERT as feature-based, mean([0:]) embedding
BERT-large, BiLSTM-CRF	90.62		word	66.6576 / -				BERT as feature-based, DSA(4, 300)
BERT-large-squad, BiLSTM	91.75	92.17	word	35.6619 / -				packed
SpanBERT-base, BiLSTM	90.46		word	30.0991 / -
SpanBERT-large, BiLSTM	91.39	92.01	word	42.5959 / -
ALBERT-base, BiLSTM	88.19		word	31.0868 / -
ALBERT-xxlarge, BiLSTM	90.39		word	107.778 / -
RoBERTa-base	90.03		word	19.2503 / -
RoBERTa-large	91.83	91.90	word	28.5525 / -
BART-large, BiLSTM	90.43		word	53.3657 / -
ELECTRA-base, BiLSTM	90.98		word	22.4132 / -				packed
ELECTRA-large	91.39		word	29.5734 / -
ELMo, BiLSTM-CRF	91.78		word, pos	74.1001 / -
ELMo, BiLSTM-CRF	91.93		word, character, pos	67.6931 / -
ELMo, GloVe, BiLSTM-CRF	92.63	92.51	word, pos	74.6521 / -				packed
ELMo, GloVe, BiLSTM-CRF	92.03		word, character, pos	60.4667 / -	182.595			threads=14, packed

* GPU / CPU : Elapsed time/example(ms), GPU / CPU(pip 1.2.0), [Tesla V100 1 GPU, Intel(R) Xeon(R) Gold 5120 CPU @ 2.20GHz, 2 CPU, 14CORES/1CPU, HyperThreading]
* CONDA : conda pytorch=1.5.0
* ONNX : conda pytorch=1.5.0, onnxruntime 1.3.0
* Dynamic : conda pytorch=1.5.0, --enable_dqm
* default batch size, learning rate, n_ctx(max_seq_length) : 32, 1e-3, 180

etagger, measured by conlleval (micro F1)

	F1 (%)	(truecase) F1 (%)	Features	GPU / CPU	Etc
GloVe, BiLSTM-CRF	87.91		word	- / -
GloVe, BiLSTM-CRF	89.20		word, pos	14.9682 / 5.0336	LSTMBlockFusedCell(), threads=14
GloVe, BiLSTM-CRF	90.06		word, character, pos	15.8913 / 5.7952	LSTMBlockFusedCell(), threads=14
GloVe, BiLSTM-CRF	90.57		word, character, pos	24.6356 / 7.0887	LSTMCell(), threads=14
GloVe, BiLSTM-CRF	90.85		word, character, pos, chunk	- / -
BERT-large, BiLSTM-CRF	90.22		word	- / -	BERT as feature-based
BERT-large, GloVe, BiLSTM-CRF	91.83		word	- / -	BERT as feature-based
ELMo, GloVe, BiLSTM-CRF	91.78		word, pos	- / -
ELMo, GloVe, BiLSTM-CRF	92.38		word, character, pos	46.4205 / 295.28	threads=14
ELMo, GloVe, BiLSTM-CRF	92.43		word, character, pos, chunk	- / -
ELMo, GloVe, BiLSTM-CRF	92.83	92.10	word, character, pos, chunk	- / -	GloVe-100d
BERT-large, ELMo, GloVe, BiLSTM-CRF	92.54		word, character, pos	- / -	BERT as feature-based, GloVe-100d

CoNLL 2003(English) leaderboard, measured by span-level F1 (micro F1, same result by conlleval? Unknown!)

	F1 (%)	Etc
CNN Large + fine-tune	93.5
biaffine-ner	93.5	https://www.aclweb.org/anthology/2020.acl-main.577.pdf
GCDT + BERT-L	93.47
LSTM-CRF+ELMo+BERT+Flair	93.38
Hierarchical + BERT	93.37
Flair embeddings + Pooling	93.18
BERT Large	92.8
BERT Base	92.4
BiLSTM-CRF+ELMo	92.22

emb_class=glove, enc_class=bilstm

train

* token_emb_dim in configs/config-glove.json == 300 (ex, glove.6B.300d.txt )
$ python preprocess.py --data_dir=data/conll2003
* --use_crf for adding crf layer, --embedding_trainable for fine-tuning pretrained word embedding
$ python train.py --data_dir=data/conll2003 --use_crf
* tensorboardX
$ rm -rf runs
$ tensorboard --logdir runs/ --port ${port} --bind_all

evaluation

$ python evaluate.py --data_dir=data/conll2003 --use_crf
* seqeval.metrics supports IOB2(BIO) format, so FB1 from conlleval.pl should be similar value with.
$ cd data/conll2003; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8864423333037378, 3684
INFO:__main__:[Elapsed Time] : 97879ms, 26.54547922888949ms on average
accuracy: 97.60%; precision: 88.90%; recall: 88.39%; FB1: 88.64
* --use_char_cnn --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.9013611454834718, 3684
INFO:__main__:[Elapsed Time] : 96906ms, 26.280749389084985ms on average
accuracy: 97.93%; precision: 89.99%; recall: 90.28%; FB1: 90.14
* --data_dir=data/conll2003_truecase --use_char_cnn --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.902609464838567, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 95198.03929328918ms, 25.81831880021024ms on average
accuracy: 97.81%; precision: 90.19%; recall: 90.33%; FB1: 90.26

emb_class=glove, enc_class=densenet

train

* token_emb_dim in configs/config-glove.json == 300 (ex, glove.6B.300d.txt )
$ python preprocess.py --config=configs/config-densenet.json --data_dir=data/conll2003
* --use_crf for adding crf layer, --embedding_trainable for fine-tuning pretrained word embedding
$ python train.py --config=configs/config-densenet.json --data_dir=data/conll2003 --save_path=pytorch-model-densenet.pt --use_crf --warmup_epoch=13 --lr_decay_rate=0.8 --epoch=64

evaluation

$ python evaluate.py --config=configs/config-densenet.json --data_dir=data/conll2003 --model_path=pytorch-model-densenet.pt --use_crf
$ cd data/conll2003; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8823112371499469, 3684
INFO:__main__:[Elapsed Time] : 91415ms, 24.789302199294053ms on average
accuracy: 97.53%; precision: 88.33%; recall: 88.14%; FB1: 88.23
INFO:__main__:[F1] : 0.8736429969745506, 3684
INFO:__main__:[Elapsed Time] : 96335ms, 26.12136844963345ms on average
accuracy: 97.36%; precision: 87.82%; recall: 86.92%; FB1: 87.36
* --use_char_cnn
INFO:__main__:[F1] : 0.8847577092511013, 3684
INFO:__main__:[Elapsed Time] : 105399ms, 28.585120825414066ms on average
accuracy: 97.60%; precision: 88.06%; recall: 88.90%; FB1: 88.48
INFO:__main__:[F1] : 0.8889086226800461, 3684
INFO:__main__:[Elapsed Time] : 103641ms, 28.099375509095847ms on average
accuracy: 97.75%; precision: 89.17%; recall: 88.62%; FB1: 88.89

emb_class=bert, enc_class=bilstm

train

* n_ctx size should be less than 512
$ python preprocess.py --config=configs/config-bert.json --data_dir=data/conll2003 --bert_model_name_or_path=./embeddings/bert-large-cased
* --use_crf for adding crf layer
* --bert_use_pos for adding Part-Of-Speech features
* --bert_use_feature_based for feature-based
* --bert_disable_lstm for removing lstm layer
$ python train.py --config=configs/config-bert.json --data_dir=data/conll2003 --save_path=pytorch-model-bert.pt --bert_model_name_or_path=./embeddings/bert-large-cased --bert_output_dir=bert-checkpoint --batch_size=16 --lr=1e-5 --epoch=10

evaluation

$ python evaluate.py --config=configs/config-bert.json --data_dir=data/conll2003 --model_path=pytorch-model-bert.pt --bert_output_dir=bert-checkpoint
$ cd data/conll2003; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.9131544214694237, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 148789ms, 40.358131957643224ms on average
accuracy: 98.27%; precision: 90.76%; recall: 91.87%; FB1: 91.32
* --batch_size=32
INFO:__main__:[F1] : 0.9118733509234828, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 141879ms, 38.48330165625848ms on average
accuracy: 98.24%; precision: 90.60%; recall: 91.78%; FB1: 91.19
* --bert_use_pos
INFO:__main__:[F1] : 0.9111325554873234, 3684
INFO:__main__:[Elapsed Time] : 141093ms, 38.29885993485342ms on average
accuracy: 98.24%; precision: 90.30%; recall: 91.94%; FB1: 91.11
* --use_crf
INFO:__main__:[F1] : 0.9058430130235833, 3684
INFO:__main__:[Elapsed Time] : 218823ms, 59.398208469055376ms on average
accuracy: 98.12%; precision: 90.44%; recall: 91.13%; FB1: 90.78
* --data_dir=data/conll2003_truecase INFO:__main__:[F1] : 0.9188571428571428, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 120905.13277053833ms, 32.785276922556875ms on average
accuracy: 98.36%; precision: 91.25%; recall: 92.53%; FB1: 91.89
* --bert_model_name_or_path=./embedings/bert-base-uncased --bert_do_lower_case --use_crf (BERT-base BiLSTM-CRF)
INFO:__main__:[F1] : 0.8993429697766097, 3684
INFO:__main__:[Elapsed Time] : 100 examples, 4368ms, 42.64646464646464ms on average
accuracy: 97.87%; precision: 89.52%; recall: 90.88%; FB1: 90.20
* --bert_model_name_or_path=./embedings/bert-base-uncased --bert_do_lower_case (BERT-base BiLSTM)
INFO:__main__:[F1] : 0.9054532577903682, 3684
INFO:__main__:[Elapsed Time] : 100 examples, 1922ms, 18.232323232323232ms on average
accuracy: 98.00%; precision: 90.55%; recall: 90.55%; FB1: 90.55
* --bert_model_name_or_path=./embedings/bert-base-uncased --bert_do_lower_case --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --epoch=20 (BERT-base BiLSTM)
INFO:__main__:[F1] : 0.9049717912552891, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 68218ms, 18.403203909856096ms on average
accuracy: 97.97%; precision: 90.12%; recall: 90.88%; FB1: 90.50
* --bert_model_name_or_path=./embedings/bert-base-uncased --bert_do_lower_case --bert_disable_lstm --use_crf (BERT-base CRF)
INFO:__main__:[F1] : 0.8961356880573526, 3684
INFO:__main__:[Elapsed Time] : 135607ms, 36.68938365462938ms on average
accuracy: 97.78%; precision: 89.24%; recall: 90.74%; FB1: 89.98
* --bert_model_name_or_path=./embedings/bert-base-uncased --bert_do_lower_case --bert_disable_lstm (BERT-base)
INFO:__main__:[F1] : 0.9024668598015978, 3684
INFO:__main__:[Elapsed Time] : 61914ms, 16.68775454792289ms on average
accuracy: 98.01%; precision: 89.50%; recall: 91.01%; FB1: 90.25
* --bert_model_name_or_path=./embedings/pytorch.uncased_L-8_H-512_A-8 --bert_do_lower_case
INFO:__main__:[F1] : 0.8829307140960977, 3684
INFO:__main__:[Elapsed Time] : 99730ms, 27.048601683410265ms on average
accuracy: 97.61%; precision: 88.25%; recall: 88.33%; FB1: 88.29
* --bert_model_name_or_path=./embedings/pytorch.uncased_L-4_H-512_A-8 --bert_do_lower_case
INFO:__main__:[F1] : 0.8634692805881324, 3684
INFO:__main__:[Elapsed Time] : 83385ms, 22.60874287265816ms on average
accuracy: 97.23%; precision: 85.38%; recall: 87.34%; FB1: 86.35
* --bert_model_name_or_path=./embedings/pytorch.uncased_L-4_H-256_A-4 --bert_do_lower_case
INFO:__main__:[F1] : 0.8155101324677603, 3684
INFO:__main__:[Elapsed Time] : 79173ms, 21.463209340211783ms on average
accuracy: 96.32%; precision: 80.82%; recall: 82.29%; FB1: 81.55
* --bert_model_name_or_path=./embedings/pytorch.uncased_L-2_H-128_A-2 --bert_do_lower_case
INFO:__main__:[F1] : 0.6965218958370878, 3684
INFO:__main__:[Elapsed Time] : 74261ms, 20.137659516698342ms on average
accuracy: 94.12%; precision: 70.92%; recall: 68.43%; FB1: 69.65
* --config=configs/config-distilbert.json --bert_model_name_or_path=./embeddings/distilbert-base-uncased --bert_do_lower_case
INFO:__main__:[F1] : 0.894963522897073, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 49652ms, 13.456421395601412ms on average
accuracy: 97.80%; precision: 88.86%; recall: 90.14%; FB1: 89.50
* --bert_model_name_or_path=./embeddings/bert-large-cased-whole-word-masking-finetuned-squad
INFO:__main__:[F1] : 0.9130013221683562, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 130683ms, 35.44013032853652ms on average
accuracy: 98.29%; precision: 90.91%; recall: 91.70%; FB1: 91.30
* --bert_model_name_or_path=./embeddings/bert-large-cased-whole-word-masking-finetuned-squad --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --epoch=30
INFO:__main__:[F1] : 0.9175393822054034, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 131497ms, 35.661960358403476ms on average
accuracy: 98.33%; precision: 91.22%; recall: 92.30%; FB1: 91.75
* --data_dir=data/conll2003_truecase --bert_model_name_or_path=./embeddings/bert-large-cased-whole-word-masking-finetuned-squad --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --epoch=30
INFO:__main__:[F1] : 0.9217421785995968, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 121090.73448181152ms, 32.83806317143229ms on average
accuracy: 98.32%; precision: 91.25%; recall: 93.11%; FB1: 92.17
* for using SpanBERT embedding, just replace pretrained BERT model to SpanBERT.
* --bert_model_name_or_path=./embeddings/spanbert_hf_base
INFO:__main__:[F1] : 0.9046450482033305, 3684
INFO:__main__:[Elapsed Time] : 110977ms, 30.09910399131143ms on average
accuracy: 98.02%; precision: 89.57%; recall: 91.38%; FB1: 90.46
* --bert_model_name_or_path=./embeddings/spanbert_hf_large
INFO:__main__:[F1] : 0.9139340659340659, 3684
INFO:__main__:[Elapsed Time] : 157069ms, 42.59598153679066ms on average
accuracy: 98.23%; precision: 90.76%; recall: 92.03%; FB1: 91.39
* --data_dir=data/conll2003_truecase --bert_model_name_or_path=./embeddings/spanbert_hf_large --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --epoch=30
INFO:__main__:[F1] : 0.9201266713581985, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 116231.20188713074ms, 31.509835921387747ms on average
accuracy: 98.35%; precision: 91.43%; recall: 92.60%; FB1: 92.01
* --bert_use_feature_based --epoch=64 --lr=1e-4 , modify model.py to use initial embedding
INFO:__main__:[F1] : 0.8610818405338954, 3684
INFO:__main__:[Elapsed Time] : 181867ms, 49.310344827586206ms on average
accuracy: 97.43%; precision: 85.42%; recall: 86.81%; FB1: 86.11
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use initial embedding
INFO:__main__:[F1] : 0.8575426322693486, 3684
INFO:__main__:[Elapsed Time] : 232758ms, 63.13765951669834ms on average
accuracy: 97.06%; precision: 86.06%; recall: 86.81%; FB1: 86.43
* --bert_use_feature_based --epoch=64 --lr=3e-4 , modify model.py to use initial+first+last embedding
INFO:__main__:[F1] : 0.8971681415929202, 3684
INFO:__main__:[Elapsed Time] : 176851ms, 47.970404561498775ms on average
accuracy: 98.00%; precision: 89.69%; recall: 89.75%; FB1: 89.72
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use initial+first+last embedding
INFO:__main__:[F1] : 0.8946078431372549, 3684
INFO:__main__:[Elapsed Time] : 247716ms, 67.20418137387999ms on average
accuracy: 97.81%; precision: 89.46%; recall: 90.47%; FB1: 89.96
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use last embedding
INFO:__main__:[F1] : 0.8912471496228732, 3684
INFO:__main__:[Elapsed Time] : 253449ms, 68.75481944067336ms on average
accuracy: 97.75%; precision: 89.38%; recall: 89.96%; FB1: 89.67
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use [-4:] embedding
INFO:__main__:[F1] : 0.90263319044703, 3684
INFO:__main__:[Elapsed Time] : 235691ms, 63.9397230518599ms on average
accuracy: 97.99%; precision: 89.94%; recall: 91.34%; FB1: 90.64
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use mean([0:3] + [-4:]) embedding
INFO:__main__:[F1] : 0.9026813186813187, 3684
INFO:__main__:[Elapsed Time] : 261161ms, 70.83220200923161ms on average
accuracy: 98.02%; precision: 90.15%; recall: 90.90%; FB1: 90.52
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use mean([0:17]) embedding
INFO:__main__:[F1] : 0.9050141242937852, 3684
INFO:__main__:[Elapsed Time] : 253013ms, 68.61390171056205ms on average
accuracy: 98.06%; precision: 90.87%; recall: 90.76%; FB1: 90.81
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use max([0:17]) embedding
INFO:__main__:[F1] : 0.9054972205064854, 3684
INFO:__main__:[Elapsed Time] : 224124ms, 60.80396415965246ms on average
accuracy: 98.09%; precision: 90.67%; recall: 90.85%; FB1: 90.76
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use mean([0:]) embedding
INFO:__main__:[F1] : 0.9071163694155041, 3684
INFO:__main__:[Elapsed Time] : 217202ms, 58.91121368449633ms on average
accuracy: 98.13%; precision: 91.00%; recall: 90.95%; FB1: 90.98
* --bert_use_feature_based --use_crf --epoch=64 --lr=3e-4 , modify model.py to use DSA(4, 300)
INFO:__main__:[F1] : 0.9036879808967896, 3684
INFO:__main__:[Elapsed Time] : 245729ms, 66.65761607385284ms on average
accuracy: 98.08%; precision: 90.78%; recall: 90.46%; FB1: 90.62
* --bert_use_feature_based --use_crf --epoch=64 --lr=1e-3 --lr_decay_rate=0.9 , modify model.py to use DSA(2, 1024)
INFO:__main__:[F1] : 0.8953335090957026, 3684
INFO:__main__:[Elapsed Time] : 219016ms, 59.40619060548466ms on average
accuracy: 97.94%; precision: 89.37%; recall: 90.19%; FB1: 89.78
* --bert_model_name_or_path=./embedings/bert-base-uncased --bert_do_lower_case --bert_remove_layers=8,9,10,11
INFO:__main__:[F1] : 0.8902760682257781, 3684
INFO:__main__:[Elapsed Time] : 91854ms, 24.907683953298942ms on average
accuracy: 97.75%; precision: 88.42%; recall: 89.64%; FB1: 89.03
* --bert_remove_layers=12,13,14,15,16,17,18,19,20,21,22,23
INFO:__main__:[F1] : 0.8910284463894966, 3684
INFO:__main__:[Elapsed Time] : 122182ms, 33.13765951669834ms on average
accuracy: 97.80%; precision: 88.11%; recall: 90.12%; FB1: 89.10

emb_class=albert, enc_class=bilstm

train

* n_ctx size should be less than 512
$ python preprocess.py --config=configs/config-albert.json --data_dir=data/conll2003 --bert_model_name_or_path=./embeddings/albert-base-v2 --bert_do_lower_case
$ python train.py --config=configs/config-albert.json --data_dir=data/conll2003 --save_path=pytorch-model-albert.pt --bert_model_name_or_path=./embeddings/albert-base-v2 --bert_output_dir=bert-checkpoint-albert --batch_size=32 --lr=1e-5 --epoch=64 --lr_decay_rate=0.9 --bert_do_lower_case

evaluation

$ python evaluate.py --config=configs/config-albert.json --data_dir=data/conll2003 --model_path=pytorch-model-albert.pt --bert_output_dir=bert-checkpoint-albert
$ cd data/conll2003; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8821014050091632, 3684
INFO:__main__:[Elapsed Time] : 114627ms, 31.08688569101276ms on average
accuracy: 97.44%; precision: 86.93%; recall: 89.48%; FB1: 88.19
* --bert_model_name_or_path=./embeddings/albert-xxlarge-v2 --batch_size=16
INFO:__main__:[F1] : 0.9041648399824638, 3684
INFO:__main__:[Elapsed Time] : 397151ms, 107.77871300570187ms on average
accuracy: 98.06%; precision: 89.51%; recall: 91.29%; FB1: 90.39

emb_class=roberta, enc_class=bilstm

train

* n_ctx size should be less than 512
$ python preprocess.py --config=configs/config-roberta.json --data_dir=data/conll2003 --bert_model_name_or_path=./embeddings/roberta-large
$ python train.py --config=configs/config-roberta.json --data_dir=data/conll2003 --save_path=pytorch-model-roberta.pt --bert_model_name_or_path=./embeddings/roberta-large --bert_output_dir=bert-checkpoint-roberta --batch_size=16 --lr=1e-5 --epoch=10

evaluation

$ python evaluate.py --config=configs/config-roberta.json --data_dir=data/conll2003 --model_path=pytorch-model-roberta.pt --bert_output_dir=bert-checkpoint-roberta
$ cd data/conll2003; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.9119915848527349, 3684
INFO:__main__:[Elapsed Time] : 145234ms, 39.37578061363019ms on average
accuracy: 98.27%; precision: 90.31%; recall: 92.10%; FB1: 91.20
* --batch_size=32
INFO:__main__:[F1] : 0.9182367890631846, 3684
INFO:__main__:[Elapsed Time] : 147221ms, 39.92967689383654ms on average
* --bert_disable_lstm
INFO:__main__:[F1] : 0.9183817062445031, 3684
INFO:__main__:[Elapsed Time] : 105306ms, 28.55253869128428ms on average
* --use_crf
INFO:__main__:[F1] : 0.9013054830287206, 3684
INFO:__main__:[Elapsed Time] : 221208ms, 60.01221830029867ms on average
* --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.9145419377527695, 3684
INFO:__main__:[Elapsed Time] : 152493ms, 41.35270160195493ms on average
* --lr_decay_rate=0.9 --bert_use_pos
INFO:__main__:[F1] : 0.914000175330937, 3684
INFO:__main__:[Elapsed Time] : 153930ms, 41.748574531631824ms on average
* --data_dir=data/conll2003_truecase --bert_disable_lstm --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --epoch=30
INFO:__main__:[F1] : 0.9190283400809718, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 112529.90531921387ms, 30.516853094295158ms on average
accuracy: 98.30%; precision: 91.37%; recall: 92.44%; FB1: 91.90
* --bert_model_name_or_path=./embeddings/roberta-base --bert_disable_lstm
INFO:__main__:[F1] : 0.9002973587545915, 3684
INFO:__main__:[Elapsed Time] : 71015ms, 19.25033939723052ms on average

emb_class=bart, enc_class=bilstm

train

* n_ctx size should be less than 512
$ python preprocess.py --config=configs/config-bart.json --data_dir=data/conll2003 --bert_model_name_or_path=./embeddings/bart-large
$ python train.py --config=configs/config-bart.json --data_dir=data/conll2003 --save_path=pytorch-model-bart.pt --bert_model_name_or_path=./embeddings/bart-large --bert_output_dir=bert-checkpoint-bart --batch_size=32 --lr=1e-5 --epoch=10

evaluation

$ python evaluate.py --config=configs/config-bart.json --data_dir=data/conll2003 --model_path=pytorch-model-bart.pt --bert_output_dir=bert-checkpoint-bart
$ cd data/conll2003; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.9042961928047503, 3684
INFO:__main__:[Elapsed Time] : 196742ms, 53.36573445560684ms on average
accuracy: 98.04%; precision: 89.21%; recall: 91.68%; FB1: 90.43

emb_class=electra, enc_class=bilstm

train

* n_ctx size should be less than 512
$ python preprocess.py --config=configs/config-electra.json --data_dir=data/conll2003 --bert_model_name_or_path=./embeddings/electra-base-discriminator --bert_do_lower_case
$ python train.py --config=configs/config-electra.json --data_dir=data/conll2003 --save_path=pytorch-model-electra.pt --bert_model_name_or_path=./embeddings/electra-base-discriminator --bert_output_dir=bert-checkpoint-electra --batch_size=32 --lr=1e-5 --epoch=20 --bert_do_lower_case

evaluation

$ python evaluate.py --config=configs/config-electra.json --data_dir=data/conll2003 --model_path=pytorch-model-electra.pt --bert_output_dir=bert-checkpoint-electra
$ cd data/conll2003; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.9072818526019194, 3684
INFO:__main__:[Elapsed Time] : 106594ms, 28.80966603312517ms on average
accuracy: 98.08%; precision: 90.24%; recall: 91.22%; FB1: 90.73
* --batch_size=64 --gradient_accumulation_steps=2 --lr=8e-5
INFO:__main__:[F1] : 0.9097631833788185, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 82651ms, 22.413250067879446ms on average
accuracy: 97.98%; precision: 90.47%; recall: 91.48%; FB1: 90.98
* --bert_model_name_or_path=./embeddings/electra-large-discriminator --lr=1e-6 --bert_disable_lstm --epoch=40
INFO:__main__:[F1] : 0.91392938696645, 3684
INFO:__main__:[Elapsed Time] : 109367ms, 29.573445560684224ms on average
* --bert_model_name_or_path=./embeddings/electra-large-discriminator --bert_disable_lstm --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --gradient_accumulation_steps=2 --lr=8e-5 --epoch=40
INFO:__main__:[F1] : 0.9042280872098155, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 116181ms, 31.50203638338311ms on average
accuracy: 98.04%; precision: 90.16%; recall: 90.69%; FB1: 90.42

emb_class=elmo, enc_class=bilstm

train

* token_emb_dim in configs/config-elmo.json == 300 (ex, glove.6B.300d.txt )
* elmo_emb_dim in configs/config-elmo.json == 1024 (ex, elmo_2x4096_512_2048cnn_2xhighway_5.5B_* )
$ python preprocess.py --config=configs/config-elmo.json --data_dir=data/conll2003 --embedding_path=embeddings/glove.6B.300d.txt
* --use_crf for adding crf layer, --embedding_trainable for fine-tuning pretrained word embedding
$ python train.py --config=configs/config-elmo.json --data_dir=data/conll2003 --save_path=pytorch-model-elmo.pt --use_crf

evaluation

$ python evaluate.py --config=configs/config-elmo.json --data_dir=data/conll2003 --model_path=pytorch-model-elmo.pt --use_crf
$ cd data/conll2003; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.9219494967331803, 3684
INFO:__main__:[Elapsed Time] : 239919ms, 65.12459283387622ms on average
accuracy: 98.29%; precision: 91.95%; recall: 92.44%; FB1: 92.19
* --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.922342119228728, 3684
INFO:__main__:[Elapsed Time] : 294772ms, 79.98968232419223ms on average
accuracy: 98.35%; precision: 92.15%; recall: 92.32%; FB1: 92.23
* --lr_decay_rate=0.9 --batch_size=64
INFO:__main__:[F1] : 0.926332565964229, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 275097ms, 74.65218571816455ms on average
accuracy: 98.45%; precision: 92.65%; recall: 92.62%; FB1: 92.63
* --data_dir=data/conll2003_truecase --batch_size=64 --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0
INFO:__main__:[F1] : 0.9251124636147129, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 236348.47974777222ms, 64.13688945796157ms on average
accuracy: 98.40%; precision: 92.18%; recall: 92.85%; FB1: 92.51
* --use_char_cnn
INFO:__main__:[F1] : 0.9137136782423814, 3684
INFO:__main__:[Elapsed Time] : 280253ms, 76.05566114580505ms on average
accuracy: 98.15%; precision: 91.44%; recall: 91.31%; FB1: 91.37
* --use_char_cnn --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.9196578181497487, 3684
INFO:__main__:[Elapsed Time] : 291372ms, 79.06489275047515ms on average
accuracy: 98.26%; precision: 91.62%; recall: 92.32%; FB1: 91.97
* --use_char_cnn --lr_decay_rate=0.9 --batch_size=64
INFO:__main__:[F1] : 0.9202508169213106, 3684
INFO:__main__:[Elapsed Time] : 3684 examples, 222863ms, 60.46673907140918ms on average
accuracy: 98.32%; precision: 91.81%; recall: 92.25%; FB1: 92.03
* --lr_decay_rate=0.9, modify model.py for disabling glove
INFO:__main__:[F1] : 0.917751217352811, 3684
INFO:__main__:[Elapsed Time] : 273089ms, 74.10019006244909ms on average
accuracy: 98.27%; precision: 91.78%; recall: 91.77%; FB1: 91.78
* --use_char_cnn --lr_decay_rate=0.9, modify model.py for disabling glove
INFO:__main__:[F1] : 0.9193262411347518, 3684
INFO:__main__:[Elapsed Time] : 249467ms, 67.69318490361118ms on average
accuracy: 98.31%; precision: 92.06%; recall: 91.80%; FB1: 91.93

Naver NER 2019 (Korean)

experiments summary

clova2019(eoj-based)

ntagger, measured by conlleval.pl (micro F1)

	F1 (%)	Features	GPU / CPU	Etc
bpe BERT(v1), BiLSTM-CRF	86.11	eoj	53.1818 / -	packed
bpe BERT(v1), BiLSTM	86.37	eoj	21.3232 / -	packed
bpe BERT(v1), CRF	86.42	eoj	35.2222 / -
bpe BERT(v1)	87.13	eoj	16.2121 / -
bpe BERT-large	85.99	eoj	30.7513 / -
bpe DistilBERT(v1)	85.30	eoj	9.0702 / -
KoELECTRA-Base	86.64	eoj	15.1616 / -
bpe ELECTRA-base(30k-512-1m)	83.09	eoj	15.8618 / -

HanBert-NER, KoELECTRA, measured by seqeval (micro F1)

(update) max_seq_len=50	F1 (%)	Features
BiLSTM-CRF	74.57	eoj
Bert-multilingual	84.20	eoj
DistilKoBERT	84.13	eoj
KoBERT	86.11	eoj
HanBert	87.31	eoj
KoELECTRA-Base	86.87	eoj
KoELECTRA-Base-v2	87.02	eoj

* note that F1 score from the 'seqeval' package for 'max_seq_len=50' might be similar with that for 'max_seq_len=180'. however, the final evaluation using 'conlleval.pl' should be different.
for example, with n_ctx=50. the F1 score from 'seqeval' is 0.8602. the F1 score from 'conlleval.pl is 0.8438.
--------------------------------------------------------------------------
'--bert_disable_lstm --lr_decay_rate=0.9 , n_ctx=50'
INFO:__main__:[F1] : 0.8602524268436113, 9000
INFO:__main__:[Elapsed Time] : 192653ms, 21.39648849872208ms on average
accuracy: 93.22%; precision: 85.55%; recall: 83.25%; FB1: 84.38
'--bert_disable_lstm --lr_decay_rate=0.9 , without --use_crf (bpe BERT), n_ctx=180'
INFO:__main__:[F1] : 0.8677214324767633, 9000
INFO:__main__:[Elapsed Time] : 868094ms, 96.45471719079897ms on average
accuracy: 94.47%; precision: 87.02%; recall: 86.33%; FB1: 86.68
--------------------------------------------------------------------------
this is due to the test.txt data has longer sequences than the 'n_ctx' size.
therefore, the evaluation results using 'seqeval' are not the final F1 score.
we recommend to use 'conlleval.pl' script for NER results.

clova2019_morph(morph-based)

ntagger, measured by conlleval.pl (micro F1)

	m-by-m F1 (%)	e-by-e F1 (%)	Features	GPU / CPU	Etc
GloVe, BiLSTM-CRF	84.29	84.29	morph, pos	30.0968 / -
GloVe, BiLSTM-CRF	85.82	85.82	morph, character, pos	25.9623 / -	packed
GloVe, DenseNet-CRF	83.44	83.49	morph, pos	25.8059 / -
GloVe, DenseNet-CRF	83.96	83.98	morph, character, pos	28.4051 / -
dha BERT(v1), BiLSTM-CRF	84.95	85.25	morph, pos	42.1063 / -	packed
dha BERT(v1), BiLSTM	84.51	85.55	morph, pos	18.9292 / -	packed
dha BERT(v1), CRF	82.94	84.99	morph, pos	46.2323 / -
dha BERT(v1)	81.15	84.26	morph, pos	15.1717 / -
dha BERT(v1), BiLSTM-CRF	83.55	83.85	morph, pos	46.0254 / -	del 8,9,10,11
dha DistilBERT(v1), CRF	79.88	82.27	morph, pos	40.2669 / -
dha DistilBERT(v1), LSTM	82.79	83.71	morph, pos	19.8174 / -
dha-bpe BERT, BiLSTM-CRF	82.83	83.83	morph, pos	42.4347 / -
dha-bpe BERT-large, BiLSTM-CRF	82.86	84.91	morph, pos	53.6760 / -
dha BERT(v2), BiLSTM-CRF	83.29	83.57	morph, pos	44.4813 / -
ELMo, BiLSTM-CRF	85.64	85.66	morph, pos	95.9868 / -
ELMo, BiLSTM-CRF	85.81	85.82	morph, character, pos	95.6196 / -
ELMo, GloVe, BiLSTM-CRF	86.37	86.37	morph, pos	82.7731 / -
ELMo, GloVe, BiLSTM-CRF	86.62	86.63	morph, character, pos	105.739 / -	packed

etagger, measured by conlleval (micro F1)

	m-by-m F1 (%)	e-by-e F1 (%)	Features	Etc
GloVe, BiLSTM-CRF	85.51	85.51	morph, character, pos
dha BERT(v1), BiLSTM-C RF	81.25	81.39	morph, pos	BERT as feature-based
ELMo, GloVe, BiLSTM-CRF	86.75	86.75	morph, character, pos

clova2019_morph_space(morph-based + space as token)

ntagger, measured by conlleval.pl (micro F1)

	m-by-m F1 (%)	e-by-e F1 (%)	Features	GPU / CPU	Etc
GloVe, BiLSTM-CRF	85.59	85.72	morph, character, pos	29.0723 / -	packed
dha BERT(v1), BiLSTM-CRF	85.17	85.61	morph, pos	43.7969 / -	packed
ELMo, GloVe, BiLSTM-CRF	85.95	86.06	morph, character, pos	113.177 / -	/packed

emb_class=glove, enc_class=bilstm

train

* for clova2019_morph
** token_emb_dim in configs/config-glove.json == 300 (ex, kor.glove.300k.300d.txt )
$ python preprocess.py --data_dir data/clova2019_morph --embedding_path embeddings/kor.glove.300k.300d.txt
** --use_crf for adding crf layer, --embedding_trainable for fine-tuning pretrained word embedding.
$ python train.py --save_path=pytorch-model-glove-kor-clova-morph.pt --data_dir data/clova2019_morph --use_crf --embedding_trainable
* for clova2019_morph_space
$ python preprocess.py --data_dir data/clova2019_morph_space --embedding_path embeddings/kor.glove.300k.300d.txt
$ python train.py --save_path=pytorch-model-glove-kor-clova-morph-space.pt --data_dir data/clova2019_morph_space --use_crf --embedding_trainable --use_char_cnn

evaluation

* for clova2019_morph
$ python evaluate.py --model_path=pytorch-model-glove-kor-clova-morph.pt --data_dir data/clova2019_morph --use_crf
** seqeval.metrics supports IOB2(BIO) format, so FB1 from conlleval.pl should be similar value with.
$ cd data/clova2019_morph; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8434531044045398, 9000
INFO:__main__:[Elapsed Time] : 270872ms, 30.096888888888888ms on average
accuracy: 93.80%; precision: 84.82%; recall: 83.76%; FB1: 84.29
*** evaluation eoj-by-eoj
$ cd data/clova2019_morph ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 93.37%; precision: 84.83%; recall: 83.76%; FB1: 84.29
** --use_char_cnn
INFO:__main__:[F1] : 0.856091088091773, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 225223ms, 25.014668296477385ms on average
accuracy: 94.23%; precision: 86.11%; recall: 84.99%; FB1: 85.55
*** evaluation eoj-by-eoj
accuracy: 93.88%; precision: 86.11%; recall: 85.00%; FB1: 85.55
** --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --use_char_cnn
INFO:__main__:[F1] : 0.8588128417407997, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 233760ms, 25.962329147683075ms on average
accuracy: 94.36%; precision: 86.48%; recall: 85.17%; FB1: 85.82
*** evaluation eoj-by-eoj
accuracy: 94.01%; precision: 86.48%; recall: 85.17%; FB1: 85.82
* for clova2019_morph_space
$ python evaluate.py --model_path=pytorch-model-glove-kor-clova-morph-space.pt --data_dir data/clova2019_morph_space --use_crf --use_char_cnn
$ cd data/clova2019_morph_space; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8573393391328268, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 261744ms, 29.072341371263473ms on average
accuracy: 95.38%; precision: 86.43%; recall: 84.77%; FB1: 85.59
*** evaluation eoj-by-eoj
$ cd data/clova2019_morph_space ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 93.89%; precision: 86.56%; recall: 84.89%; FB1: 85.72

emb_class=glove, enc_class=densenet

train

* token_emb_dim in configs/config-glove.json == 300 (ex, kor.glove.300k.300d.txt )
$ python preprocess.py --config=configs/config-densenet.json --data_dir data/clova2019_morph --embedding_path embeddings/kor.glove.300k.300d.txt
* --use_crf for adding crf layer, --embedding_trainable for fine-tuning pretrained word embedding.
$ python train.py --config=configs/config-densenet.json --save_path=pytorch-model-densenet-kor-clova-morph.pt --data_dir data/clova2019_morph --use_crf --embedding_trainable

evaluation

$ python evaluate.py --config=configs/config-densenet.json --model_path=pytorch-model-densenet-kor-clova-morph.pt --data_dir data/clova2019_morph --use_crf
* seqeval.metrics supports IOB2(BIO) format, so FB1 from conlleval.pl should be similar value with.
$ cd data/clova2019_morph; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8350127432612621, 9000
INFO:__main__:[Elapsed Time] : 232331ms, 25.805978442049117ms on average
accuracy: 93.42%; precision: 82.80%; recall: 84.10%; FB1: 83.44
** evaluation eoj-by-eoj
$ cd data/clova2019_morph ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 92.96%; precision: 82.86%; recall: 84.13%; FB1: 83.49
* --user_char_cnn --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.8402205267380136, 9000
INFO:__main__:[Elapsed Time] : 255785ms, 28.405156128458717ms on average
accuracy: 93.66%; precision: 84.25%; recall: 83.68%; FB1: 83.96
** evaluation eoj-by-eoj
accuracy: 93.24%; precision: 84.28%; recall: 83.69%; FB1: 83.98

emb_class=bert, enc_class=bilstm, bpe BERT(v1), bpe BERT-large, dha BERT(v1)

train

* n_ctx size should be less than 512
* for clova2019
$ python preprocess.py --config=configs/config-bert.json --data_dir data/clova2019 --bert_model_name_or_path=./embeddings/kor-bert-base-bpe.v1
$ python train.py --config=configs/config-bert.json --save_path=pytorch-model-bert-kor-eoj.pt --bert_model_name_or_path=./embeddings/kor-bert-base-bpe.v1 --bert_output_dir=bert-checkpoint-kor-eoj --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/clova2019 --use_crf
* for clova2019_morph
$ python preprocess.py --config=configs/config-bert.json --data_dir data/clova2019_morph --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v1
$ python train.py --config=configs/config-bert.json --save_path=pytorch-model-bert-kor-clova-morph.pt --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v1 --bert_output_dir=bert-checkpoint-kor-clova-morph --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/clova2019_morph --use_crf --bert_use_pos
* for clova2019_morph_space
$ python preprocess.py --config=configs/config-bert.json --data_dir data/clova2019_morph_space --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v1
$ python train.py --config=configs/config-bert.json --save_path=pytorch-model-bert-kor-clova-morph-space.pt --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v1 --bert_output_dir=bert-checkpoint-kor-clova-morph-space --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/clova2019_morph_space --use_crf --bert_use_pos

evaluation


* for clova2019
$ python evaluate.py --config=configs/config-bert.json --model_path=pytorch-model-bert-kor-eoj.pt --data_dir data/clova2019 --bert_output_dir=bert-checkpoint-kor-eoj --use_crf
$ cd data/clova2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
(bpe BERT BiLSTM-CRF)
INFO:__main__:[F1] : 0.8613367390378885, 9000
INFO:__main__:[Elapsed Time] : 476814ms, 52.96955217246361ms on average
accuracy: 94.15%; precision: 86.24%; recall: 85.99%; FB1: 86.11
** --lr_decay_rate=0.9 , without --use_crf (bpe BERT BiLSTM)
INFO:__main__:[F1] : 0.8646059046587216, 9000
INFO:__main__:[Elapsed Time] : 100 examples, 2224ms, 21.32323232323232ms on average
accuracy: 94.31%; precision: 85.92%; recall: 86.82%; FB1: 86.37 ** --bert_disable_lstm --lr_decay_rate=0.9 (bpe BERT CRF)
INFO:__main__:[F1] : 0.8643569376373161, 9000
INFO:__main__:[Elapsed Time] : 342154ms, 38.00722302478053ms on average
accuracy: 94.35%; precision: 85.90%; recall: 86.94%; FB1: 86.42
** --bert_disable_lstm --lr_decay_rate=0.9 , without --use_crf (bpe BERT)
INFO:__main__:[F1] : 0.8677214324767633, 9000
INFO:__main__:[Elapsed Time] : 868094ms, 96.45471719079897ms on average
accuracy: 94.47%; precision: 87.02%; recall: 86.33%; FB1: 86.68
** --bert_disable_lstm --lr_decay_rate=0.9 , n_ctx=50
INFO:__main__:[F1] : 0.8602524268436113, 9000
INFO:__main__:[Elapsed Time] : 192653ms, 21.39648849872208ms on average
accuracy: 93.22%; precision: 85.55%; recall: 83.25%; FB1: 84.38
** --bert_disable_lstm --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --lr=8e-5 --epoch=30 , without --use_crf (bpe BERT)
INFO:__main__:[F1] : 0.863227606609181, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 889043ms, 98.78186465162796ms on average
accuracy: 94.20%; precision: 85.18%; recall: 87.30%; FB1: 86.23
** --bert_disable_lstm --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --lr=8e-5 --gradient_accumulation_steps=2 --epoch=30 , without --use_crf (bpe BERT)
INFO:__main__:[F1] : 0.8722265771446098, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 952261ms, 105.80508945438382ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 1714ms, 16.21212121212121ms on average
accuracy: 94.63%; precision: 87.25%; recall: 87.01%; FB1: 87.13
** --bert_model_name_or_path=./embeddings/kor-bert-large-bpe --bert_disable_lstm --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --lr=8e-5 --gradient_accumulation_steps=2 --epoch=30 , without --use_crf (bpe BERT-large) INFO:__main__:[F1] : 0.8608467232968307, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 1040116.376876831ms, 115.56598331838438ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 3212.3892307281494ms, 30.75131984672161ms on average
accuracy: 94.13%; precision: 86.19%; recall: 85.79%; FB1: 85.99
** --config=configs/config-distilbert.json --bert_model_name_or_path=./embeddings/kor-distil-bpe-bert.v1 --bert_disable_lstm --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --lr=8e-5 --gradient_accumulation_steps=2 --epoch=30 , without --use_crf
INFO:__main__:[F1] : 0.852160598843144, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 820104.8340797424ms, 91.12164719606297ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 1019.8376178741455ms, 9.317352314188023ms on average
accuracy: 93.87%; precision: 85.10%; recall: 85.15%; FB1: 85.12
INFO:__main__:[F1] : 0.85393906493859, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 816257.2820186615ms, 90.68703071211878ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 988.6960983276367ms, 9.070220619741113ms on average
accuracy: 93.97%; precision: 85.03%; recall: 85.57%; FB1: 85.30
* for clova2019_morph
$ python evaluate.py --config=configs/config-bert.json --model_path=pytorch-model-bert-kor-clova-morph.pt --data_dir=data/clova2019_morph --bert_output_dir=bert-checkpoint-kor-clova-morph --use_crf --bert_use_pos
$ cd data/clova2019_morph; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
(dha BERT BiLSTM-CRF)
INFO:__main__:[F1] : 0.850251256281407, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 379036ms, 42.10634514946105ms on average
accuracy: 94.30%; precision: 85.06%; recall: 84.84%; FB1: 84.95
*** evaluation eoj-by-eoj
$ cd data/clova2019_morph ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 93.86%; precision: 85.50%; recall: 84.99%; FB1: 85.25
** --lr_decay_rate=0.9 , without --use_crf (dha BERT BiLSTM)
INFO:__main__:[F1] : 0.8453688900983027, 9000
INFO:__main__:[Elapsed Time] : 100 examples, 2197ms, 21.11111111111111ms on average
accuracy: 94.39%; precision: 84.43%; recall: 84.49%; FB1: 84.46
*** evaluation eoj-by-eoj
accuracy: 93.94%; precision: 85.90%; recall: 84.93%; FB1: 85.41 ** --bert_disable_lstm --lr_decay_rate=0.9 (dha BERT CRF)
INFO:__main__:[F1] : 0.8300803673938002, 9000
INFO:__main__:[Elapsed Time] : 100 examples, 4709ms, 46.23232323232323ms on average
accuracy: 94.18%; precision: 82.33%; recall: 83.57%; FB1: 82.94
*** evaluation eoj-by-eoj
accuracy: 93.84%; precision: 85.15%; recall: 84.83%; FB1: 84.99
** --bert_disable_lstm --lr_decay_rate=0.9 , without --use_crf (dha BERT)
INFO:__main__:[F1] : 0.8122244286627849, 9000
INFO:__main__:[Elapsed Time] : 100 examples, 1604ms, 15.171717171717171ms on average
accuracy: 93.85%; precision: 80.28%; recall: 82.04%; FB1: 81.15
*** evaluation eoj-by-eoj
accuracy: 93.48%; precision: 84.92%; recall: 83.61%; FB1: 84.26
** bert_outputs[2][-7], --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.8296454550078846, 9000
INFO:__main__:[Elapsed Time] : 376186ms, 41.786642960328926ms on average
accuracy: 93.73%; precision: 82.62%; recall: 83.17%; FB1: 82.90
*** evaluation eoj-by-eoj
accuracy: 93.19%; precision: 83.25%; recall: 83.34%; FB1: 83.29
** --bert_remove_layers=8,9,10,11
INFO:__main__:[F1] : 0.8361804271488914, 9000
INFO:__main__:[Elapsed Time] : 414339ms, 46.025447271919106ms on average
accuracy: 93.83%; precision: 83.31%; recall: 83.79%; FB1: 83.55
*** evaluation eoj-by-eoj
accuracy: 93.31%; precision: 83.78%; recall: 83.93%; FB1: 83.85 ** --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --lr=5e-5 --gradient_accumulation_steps=2 --epoch=30 , without --use_crf (dha BERT BiLSTM)
INFO:__main__:[F1] : 0.8459056275447281, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 902977ms, 100.3300366707412ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 1982ms, 18.92929292929293ms on average
accuracy: 94.48%; precision: 83.93%; recall: 85.11%; FB1: 84.51
*** evaluation eoj-by-eoj
accuracy: 94.03%; precision: 85.55%; recall: 85.55%; FB1: 85.55
** --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --lr=2e-5 --epoch=30 --bert_disable_lstm , without --use_crf (dha BERT)
INFO:__main__:[F1] : 0.8135219179456316, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 887293ms, 98.58706522946994ms on average
accuracy: 94.03%; precision: 79.90%; recall: 82.72%; FB1: 81.28
*** evaluation eoj-by-eoj
accuracy: 93.69%; precision: 84.50%; recall: 84.63%; FB1: 84.56
** --config=configs/config-distilbert.json --bert_model_name_or_path=./embeddings/kor-distil-dha-bert.v1 --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --epoch=30 --bert_disable_lstm --lr=8e-5
(1) epoch_0
INFO:__main__:[F1] : 0.7994317797470066, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 331695.18399238586ms, 36.84263667049296ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 4152.508020401001ms, 40.266978620278714ms on average
accuracy: 93.10%; precision: 79.41%; recall: 80.35%; FB1: 79.88
*** evaluation eoj-by-eoj
accuracy: 92.69%; precision: 82.73%; recall: 81.82%; FB1: 82.27
** --config=configs/config-distilbert.json --bert_model_name_or_path=./embeddings/kor-distil-dha-bert.v1 --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --epoch=30 --lr=8e-5 , without --use_crf
(1) epoch_0
INFO:__main__:[F1] : 0.8286240267526895, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 915342.8752422333ms, 101.70127953644766ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 2091.184377670288ms, 19.81741250163377ms on average
accuracy: 93.82%; precision: 82.19%; recall: 83.40%; FB1: 82.79
*** evaluation eoj-by-eoj
accuracy: 93.26%; precision: 83.69%; recall: 83.73%; FB1: 83.71
* for clova2019_morph_space
$ python evaluate.py --config=configs/config-bert.json --model_path=pytorch-model-bert-kor-clova-morph-space.pt --data_dir=data/clova2019_morph_space --bert_output_dir=bert-checkpoint-kor-clova-morph-space --use_crf --bert_use_pos
$ cd data/clova2019_morph_space; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.853110511030723, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 394271ms, 43.79697744193799ms on average
accuracy: 95.51%; precision: 84.96%; recall: 85.38%; FB1: 85.17
*** evaluation eoj-by-eoj
$ cd data/clova2019_morph_space ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 93.93%; precision: 85.52%; recall: 85.70%; FB1: 85.61

emb_class=bert, enc_class=bilstm, dha-bpe BERT, dha-bpe BERT-large, dha BERT(v2)

train

* n_ctx size should be less than 512
* for clova2019_morph
** dha-bpe
$ python preprocess.py --config=configs/config-bert.json --data_dir data/clova2019_morph --bert_model_name_or_path=./embeddings/kor-bert-base-dha_bpe
$ python train.py --config=configs/config-bert.json --save_path=pytorch-model-bert-kor-clova-morph.pt --bert_model_name_or_path=./embeddings/kor-bert-base-dha_bpe --bert_output_dir=bert-checkpoint-kor-clova-morph --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/clova2019_morph --use_crf --bert_use_pos
** dha
$ python preprocess.py --config=configs/config-bert.json --data_dir data/clova2019_morph --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v2
$ python train.py --config=configs/config-bert.json --save_path=pytorch-model-bert-kor-clova-morph.pt --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v2 --bert_output_dir=bert-checkpoint-kor-clova-morph --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/clova2019_morph --use_crf --bert_use_pos

evaluation

* for clova2019_morph
** dha-bpe
$ python evaluate.py --config=configs/config-bert.json --model_path=pytorch-model-bert-kor-clova-morph.pt --data_dir=data/clova2019_morph --bert_output_dir=bert-checkpoint-kor-clova-morph --use_crf --bert_use_pos
$ cd data/clova2019_morph; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8295019157088124, 9000
INFO:__main__:[Elapsed Time] : 382042ms, 42.434714968329814ms on average
accuracy: 93.77%; precision: 81.78%; recall: 83.91%; FB1: 82.83
*** evaluation eoj-by-eoj
$ cd data/clova2019_morph ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 93.37%; precision: 83.34%; recall: 84.33%; FB1: 83.83
*** --bert_model_name_or_path=./embeddings/kor-bert-large-dha_bpe --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --lr=1e-5
INFO:__main__:[F1] : 0.8298886586824331, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 483195.63341140747ms, 53.676061569842936ms on average
accuracy: 94.17%; precision: 81.99%; recall: 83.75%; FB1: 82.86
*** evaluation eoj-by-eoj
accuracy: 93.85%; precision: 84.97%; recall: 84.85%; FB1: 84.91
** dha
$ python evaluate.py --config=configs/config-bert.json --model_path=pytorch-model-bert-kor-clova-morph.pt --data_dir=data/clova2019_morph --bert_output_dir=bert-checkpoint-kor-clova-morph --use_crf --bert_use_pos
$ cd data/clova2019_morph; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8336304521299173, 9000
INFO:__main__:[Elapsed Time] : 400446ms, 44.48138682075786ms on average
accuracy: 93.58%; precision: 83.12%; recall: 83.46%; FB1: 83.29
*** evaluation eoj-by-eoj
$ cd data/clova2019_morph ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 93.06%; precision: 83.55%; recall: 83.59%; FB1: 83.57

emb_class=elmo, enc_class=bilstm

train

* for clova2019_morph
** token_emb_dim in configs/config-elmo.json == 300 (ex, kor.glove.300k.300d.txt )
** elmo_emb_dim in configs/config-elmo.json == 1024 (ex, kor_elmo_2x4096_512_2048cnn_2xhighway_1000k* )
$ python preprocess.py --config=configs/config-elmo.json --data_dir=data/clova2019_morph --embedding_path=embeddings/kor.glove.300k.300d.txt
$ python train.py --config=configs/config-elmo.json --save_path=pytorch-model-elmo-kor-clova-morph.pt --data_dir=data/clova2019_morph --elmo_options_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_options.json --elmo_weights_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_weights.hdf5 --use_crf
* for clova2019_morph_space
$ python preprocess.py --config=configs/config-elmo.json --data_dir=data/clova2019_morph_space --embedding_path=embeddings/kor.glove.300k.300d.txt
$ python train.py --config=configs/config-elmo.json --save_path=pytorch-model-elmo-kor-clova-morph-space.pt --data_dir=data/clova2019_morph_space --elmo_options_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_options.json --elmo_weights_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_weights.hdf5 --use_crf --use_char_cnn

evaluation

* for clova2019_morph
$ python evaluate.py --config=configs/config-elmo.json --model_path=pytorch-model-elmo-kor-clova-morph.pt --data_dir=data/clova2019_morph --elmo_options_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_options.json --elmo_weights_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_weights.hdf5 --use_crf
$ cd data/clova2019_morph; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
** --embedding_trainable
INFO:__main__:[F1] : 0.8642865647270933, 9000
INFO:__main__:[Elapsed Time] : 744958ms, 82.7731111111111ms on average
accuracy: 94.63%; precision: 86.36%; recall: 86.38%; FB1: 86.37
*** evluation eoj-by-eoj
$ cd data/clova2019_morph ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 94.26%; precision: 86.37%; recall: 86.38%; FB1: 86.37
** --use_char_cnn --embedding_trainable
INFO:__main__:[F1] : 0.866860266484455, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 951810ms, 105.7396377375264ms on average
accuracy: 94.65%; precision: 86.99%; recall: 86.26%; FB1: 86.62
*** evaluation eoj-by-eoj
accuracy: 94.33%; precision: 87.00%; recall: 86.26%; FB1: 86.63
** modify model.py for disabling glove
INFO:__main__:[F1] : 0.856990797967312, 9000
INFO:__main__:[Elapsed Time] : 863968ms, 95.98688743193688ms on average
accuracy: 94.30%; precision: 86.44%; recall: 84.85%; FB1: 85.64
*** evaluation eoj-by-eoj
accuracy: 93.95%; precision: 86.46%; recall: 84.87%; FB1: 85.66 ** --use_char_cnn , modify model.py for disabling glove
INFO:__main__:[F1] : 0.8587286088699316, 9000
INFO:__main__:[Elapsed Time] : 860675ms, 95.61962440271141ms on average
accuracy: 94.29%; precision: 86.42%; recall: 85.21%; FB1: 85.81
*** evaluation eoj-by-eoj
accuracy: 94.01%; precision: 86.42%; recall: 85.22%; FB1: 85.82
* for clova2019_morph_space
$ python evaluate.py --config=configs/config-elmo.json --model_path=pytorch-model-elmo-kor-clova-morph-space.pt --data_dir=data/clova2019_morph_space --elmo_options_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_options.json --elmo_weights_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_weights.hdf5 --use_crf --use_char_cnn
$ cd data/clova2019_morph_space; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8609660351595835, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 1018694ms, 113.17768640960107ms on average
accuracy: 95.51%; precision: 86.16%; recall: 85.74%; FB1: 85.95
*** evaluation eoj-by-eoj
$ cd data/clova2019_morph_space ; python to-eoj.py < test.txt.pred > test.txt.pred.eoj ; perl ../../etc/conlleval.pl < test.txt.pred.eoj ; cd ../..
accuracy: 94.08%; precision: 86.28%; recall: 85.85%; FB1: 86.06

emb_class=electra, enc_class=bilstm, KoELECTRA-Base, bpe ELECTRA-base(30k-512-1m)

train

* n_ctx size should be less than 512
* for clova2019
** KoELECTRA-Base
$ python preprocess.py --config=configs/config-electra.json --data_dir data/clova2019 --bert_model_name_or_path=./embeddings/koelectra-base-discriminator
$ python train.py --config=configs/config-electra.json --save_path=pytorch-model-bert-kor-eoj.pt --bert_model_name_or_path=./embeddings/koelectra-base-discriminator --bert_output_dir=bert-checkpoint-kor-eoj --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/clova2019 --bert_disable_lstm
** bpe ELECTRA-base(30k-512-1m)
$ python preprocess.py --config=configs/config-electra.json --data_dir data/clova2019 --bert_model_name_or_path=./embeddings/kor-electra-base-bpe-30k-512-1m
$ python train.py --config=configs/config-electra.json --save_path=pytorch-model-bert-kor-eoj.pt --bert_model_name_or_path=./embeddings/kor-electra-base-bpe-30k-512-1m --bert_output_dir=bert-checkpoint-kor-eoj --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/clova2019 --bert_disable_lstm

evaluation


* for clova2019
** KoELECTRA-Base
$ python evaluate.py --config=configs/config-electra.json --model_path=pytorch-model-bert-kor-eoj.pt --data_dir data/clova2019 --bert_output_dir=bert-checkpoint-kor-eoj --bert_disable_lstm
$ cd data/clova2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8601332716652106, 9000
INFO:__main__:[Elapsed Time] : 100 examples, 1667ms, 15.737373737373737ms on average
accuracy: 94.17%; precision: 86.46%; recall: 85.36%; FB1: 85.90
*** --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --gradient_accumulation_steps=2 --lr=8e-5 , n_ctx=50
INFO:__main__:[F1] : 0.8647250807012538, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 286224ms, 31.79564396044005ms on average
accuracy: 93.01%; precision: 85.94%; recall: 82.36%; FB1: 84.12
*** --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --gradient_accumulation_steps=2 --lr=8e-5 --epoch=30
INFO:__main__:[F1] : 0.8674485806561278, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 976734ms, 108.52672519168796ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 1597ms, 15.16161616161616ms on average
accuracy: 94.48%; precision: 86.52%; recall: 86.75%; FB1: 86.64
** bpe ELECTRA-base(30k-512-1m)
$ python evaluate.py --config=configs/config-electra.json --model_path=pytorch-model-bert-kor-eoj.pt --data_dir data/clova2019 --bert_output_dir=bert-checkpoint-kor-eoj --bert_disable_lstm
$ cd data/clova2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
*** --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --gradient_accumulation_steps=2 --lr=8e-5 --epoch=30
INFO:__main__:[F1] : 0.8319372174358101, 9000
INFO:__main__:[Elapsed Time] : 9000 examples, 900059.1132640839ms, 100.0054818893197ms on average
INFO:__main__:[Elapsed Time] : 100 examples, 1674.2236614227295ms, 15.86188692035097ms on average
accuracy: 93.08%; precision: 83.28%; recall: 82.90%; FB1: 83.09

KMOU NER 2019 (Korean)

experiments summary

ntagger, measured by conlleval.pl / token_eval.py (micro F1)

	span / token F1 (%)	Features	GPU / CPU	Etc
GloVe, BiLSTM-CRF	85.26 / 86.54	morph, pos	23.0755 / -	packed
GloVe, BiLSTM-CRF	85.93 / 86.41	morph, character, pos	27.7451 / -	packed
GloVe, DenseNet-CRF	85.30 / 86.89	morph, pos	24.0280 / -
GloVe, DenseNet-CRF	85.91 / 86.38	morph, character, pos	22.7710 / -
dha BERT(v1), BiLSTM-CRF	87.56 / 90.47	morph, pos	40.0766 / -	packed
dha BERT(v1), BiLSTM	88.00 / 90.24	morph, pos	23.0388 / -	packed
dha BERT(v1), CRF	88.46 / 90.56	morph, pos	34.1522 / -
dha BERT(v1)	88.04 / 90.64	morph, pos	17.8542 / -
dha BERT(v1), BiLSTM-CRF	83.99 / 87.54	morph, pos	40.5205 / -	del 8,9,10,11
dha DistilBERT(v1), BiLSTM-CRF	84.20 / 86.94	morph, pos	32.1762 / -
dha DistilBERT(v1), CRF	84.85 / 87.34	morph, pos	27.4700 / -
dha BERT(v2), BiLSTM-CRF	85.24 / 87.35	morph, pos	37.7829 / -
dha-bpe BERT, BiLSTM-CRF	85.18 / 88.01	morph, pos	39.0183 / -
dha-bpe BERT-large, CRF	89.02 / 91.07	morph, pos	45.1637 / -
ELMo, BiLSTM-CRF	88.22 / 89.05	morph, pos	128.029 / -
ELMo, BiLSTM-CRF	88.25 / 89.26	morph, character, pos	127.514 / -
ELMo, GloVe, BiLSTM-CRF	88.10 / 88.71	morph, pos	127.989 / -	packed
ELMo, GloVe, BiLSTM-CRF	88.00 / 89.20	morph, character, pos	116.965 / -	packed

etagger, measured by conlleval (micro F1)

	span / token F1 (%)	Features
ELMo, GloVe, BiLSTM-CRF	89.09 / 89.90	morph, character, pos

Pytorch-BERT-CRF-NER, measured by sklearn.metrics (token-level F1)

	token-level macro / micro F1 (%)	Features
KoBERT+CRF	87.56 / 89.70	morph

emb_class=glove, enc_class=bilstm

train

* token_emb_dim in configs/config-glove.json == 300 (ex, kor.glove.300k.300d.txt )
$ python preprocess.py --data_dir data/kmou2019 --embedding_path embeddings/kor.glove.300k.300d.txt
* --use_crf for adding crf layer, --embedding_trainable for fine-tuning pretrained word embedding.
$ python train.py --save_path=pytorch-model-glove-kor-kmou-morph.pt --data_dir data/kmou2019 --use_crf --embedding_trainable

evaluation

$ python evaluate.py --model_path=pytorch-model-glove-kor-kmou-morph.pt --data_dir data/kmou2019 --use_crf
* seqeval.metrics supports IOB2(BIO) format, so FB1 from conlleval.pl should be similar value with.
$ cd data/kmou2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
* token-level micro F1
$ cd data/kmou2019; python ../../etc/token_eval.py < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8526331317303437, 927
INFO:__main__:[Elapsed Time] : 927 examples, 21481ms, 23.0755939524838ms on average
accuracy: 96.79%; precision: 85.66%; recall: 84.88%; FB1: 85.26
token_eval micro F1: 0.865492518703242
* --use_char_cnn
INFO:__main__:[F1] : 0.8593403342700785, 927
INFO:__main__:[Elapsed Time] : 927 examples, 25819ms, 27.7451403887689ms on average
accuracy: 96.74%; precision: 86.56%; recall: 85.32%; FB1: 85.93
token_eval micro F1: 0.8641512381845168
* --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --use_char_cnn
INFO:__main__:[F1] : 0.8589818607372732, 927
INFO:__main__:[Elapsed Time] : 927 examples, 23254ms, 24.571274298056156ms on average
accuracy: 96.87%; precision: 85.57%; recall: 86.23%; FB1: 85.90
token_eval micro F1: 0.8703660344962149

emb_class=glove, enc_class=densenet

train

* token_emb_dim in configs/config-glove.json == 300 (ex, kor.glove.300k.300d.txt )
$ python preprocess.py --config=configs/config-densenet.json --data_dir data/kmou2019 --embedding_path embeddings/kor.glove.300k.300d.txt
* --use_crf for adding crf layer, --embedding_trainable for fine-tuning pretrained word embedding.
$ python train.py --config=configs/config-densenet.json --save_path=pytorch-model-densenet-kor-kmou-morph.pt --data_dir data/kmou2019 --use_crf --embedding_trainable

evaluation

$ python evaluate.py --config=configs/config-densenet.json --model_path=pytorch-model-densenet-kor-kmou-morph.pt --data_dir data/kmou2019 --use_crf
* seqeval.metrics supports IOB2(BIO) format, so FB1 from conlleval.pl should be similar value with.
$ cd data/kmou2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
* token-level micro F1
$ cd data/kmou2019; python ../../etc/token_eval.py < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8530318602261048, 927
INFO:__main__:[Elapsed Time] : 927 examples, 22368ms, 24.028077753779698ms on average
accuracy: 96.85%; precision: 85.29%; recall: 85.32%; FB1: 85.30
token_eval micro F1: 0.8689248895434463
* --use_char_cnn
INFO:__main__:[F1] : 0.8591299200947027, 927
INFO:__main__:[Elapsed Time] : 927 examples, 21204ms, 22.771058315334773ms on average
accuracy: 96.77%; precision: 86.58%; recall: 85.26%; FB1: 85.91
token_eval micro F1: 0.8638321196460731

emb_class=bert, enc_class=bilstm, dha BERT(v1), dha BERT(v2)

train

* n_ctx size should be less than 512
* dha (v1)
$ python preprocess.py --config=configs/config-bert.json --data_dir data/kmou2019 --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v1
$ python train.py --config=configs/config-bert.json --save_path=pytorch-model-bert-kor-kmou-morph.pt --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v1 --bert_output_dir=bert-checkpoint-kor-kmou-morph --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/kmou2019 --use_crf --bert_use_pos
* dha (v2)
$ python preprocess.py --config=configs/config-bert.json --data_dir data/kmou2019 --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v2
$ python train.py --config=configs/config-bert.json --save_path=pytorch-model-bert-kor-kmou-morph.pt --bert_model_name_or_path=./embeddings/kor-bert-base-dha.v2 --bert_output_dir=bert-checkpoint-kor-kmou-morph --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/kmou2019 --use_crf --bert_use_pos

evaluation

* dha (v1)
$ python evaluate.py --config=configs/config-bert.json --model_path=pytorch-model-bert-kor-kmou-morph.pt --data_dir=data/kmou2019 --bert_output_dir=bert-checkpoint-kor-kmou-morph --use_crf --bert_use_pos
$ cd data/kmou2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
$ cd data/kmou2019; python ../../etc/token_eval.py < test.txt.pred ; cd ../..
(dha BERT BiLSTM-CRF)
INFO:__main__:[F1] : 0.8751814223512336, 927
INFO:__main__:[Elapsed Time] : 927 examples, 37240ms, 40.076673866090715ms on average
accuracy: 97.58%; precision: 86.59%; recall: 88.55%; FB1: 87.56
token_eval micro F1: 0.9047362341162879
** --lr_decay_rate=0.9 , without --use_crf (dha BERT BiLSTM)
INFO:__main__:[F1] : 0.8800116635078, 927
INFO:__main__:[Elapsed Time] : 927 examples, 21444ms, 23.03887688984881ms on average
accuracy: 97.56%; precision: 87.38%; recall: 88.63%; FB1: 88.00
token_eval micro F1: 0.9024978600887091
** --bert_disable_lstm --lr_decay_rate=0.9 (dha BERT CRF)
INFO:__main__:[F1] : 0.8844425112367696, 927
INFO:__main__:[Elapsed Time] : 927 examples, 31752ms, 34.152267818574515ms on average
accuracy: 97.64%; precision: 87.37%; recall: 89.57%; FB1: 88.46
token_eval micro F1: 0.9056049478160032
** --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --bert_disable_lstm --epoch=30 (dha BERT CRF)
INFO:__main__:[F1] : 0.8810802962102512, 927
INFO:__main__:[Elapsed Time] : 927 examples, 30447ms, 32.355291576673864ms on average
accuracy: 97.54%; precision: 87.16%; recall: 89.10%; FB1: 88.12
token_eval micro F1: 0.8997748622001397
** --bert_disable_lstm --lr_deacy_rate=0.9 , without --use_crf (dha BERT)
INFO:__main__:[F1] : 0.880439496891716, 927
INFO:__main__:[Elapsed Time] : 927 examples, 16635ms, 17.854211663066955ms on average
accuracy: 97.65%; precision: 86.70%; recall: 89.43%; FB1: 88.04
token_eval micro F1: 0.9064737125702409
** --bert_remove_layers=8,9,10,11
INFO:__main__:[F1] : 0.8392018779342724, 927
INFO:__main__:[Elapsed Time] : 37666ms, 40.52051835853132ms on average
accuracy: 96.88%; precision: 83.99%; recall: 83.99%; FB1: 83.99
token_eval micro F1: 0.8754817902934005
** --config=configs/config-distilbert.json --bert_model_name_or_path=./embeddings/kor-distil-dha-bert.v1 --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --bert_disable_lstm --epoch=30 --lr=8e-5
(1) epoch_0
INFO:__main__:[F1] : 0.8477473562219324, 927
INFO:__main__:[Elapsed Time] : 927 examples, 25547.332048416138ms, 27.470011175579952ms on average
accuracy: 96.96%; precision: 83.79%; recall: 85.93%; FB1: 84.85
token_eval micro F1: 0.8734618063617366
** --config=configs/config-distilbert.json --bert_model_name_or_path=./embeddings/kor-distil-dha-bert.v1 --use_transformers_optimizer --warmup_epoch=0 --weight_decay=0.0 --epoch=30 --lr=8e-5
(1) epoch_2
INFO:__main__:[F1] : 0.8417381194593809, 927
INFO:__main__:[Elapsed Time] : 927 examples, 29921.59104347229ms, 32.176273945340846ms on average
accuracy: 96.85%; precision: 83.36%; recall: 85.05%; FB1: 84.20
token_eval micro F1: 0.8694366635543106
* dha(v2)
$ python evaluate.py --config=configs/config-bert.json --model_path=pytorch-model-bert-kor-kmou-morph.pt --data_dir=data/kmou2019 --bert_output_dir=bert-checkpoint-kor-kmou-morph --use_crf --bert_use_pos
$ cd data/kmou2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
$ cd data/kmou2019; python ../../etc/token_eval.py < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8519003931847969, 927
INFO:__main__:[Elapsed Time] : 35123ms, 37.7829373650108ms on average
accuracy: 96.83%; precision: 84.59%; recall: 85.90%; FB1: 85.24
token_eval micro F1: 0.8735865242143024

emb_class=bert, enc_class=bilstm, dha-bpe BERT, dha-bpe BERT-large

train

* n_ctx size should be less than 512
$ python preprocess.py --config=configs/config-bert.json --data_dir data/kmou2019 --bert_model_name_or_path=./embeddings/kor-bert-base-dha_bpe
$ python train.py --config=configs/config-bert.json --save_path=pytorch-model-bert-kor-kmou-morph.pt --bert_model_name_or_path=./embeddings/kor-bert-base-dha_bpe --bert_output_dir=bert-checkpoint-kor-kmou-morph --batch_size=32 --lr=5e-5 --epoch=20 --data_dir data/kmou2019 --use_crf --bert_use_pos

evaluation

$ python evaluate.py --config=configs/config-bert.json --model_path=pytorch-model-bert-kor-kmou-morph.pt --data_dir=data/kmou2019 --bert_output_dir=bert-checkpoint-kor-kmou-morph --use_crf --bert_use_pos
$ cd data/kmou2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
$ cd data/kmou2019; python ../../etc/token_eval.py < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8517251211861989, 927
INFO:__main__:[Elapsed Time] : 36267ms, 39.018358531317496ms on average
accuracy: 97.14%; precision: 82.79%; recall: 87.72%; FB1: 85.18
token_eval micro F1: 0.8801729462631254
* --bert_model_name_or_path=./embeddings/kor-bert-large-dha_bpe --bert_disable_lstm --lr_decay_rate=0.9 --lr=1e-5 INFO:__main__:[F1] : 0.8902403706921518, 927
INFO:__main__:[Elapsed Time] : 927 examples, 41991.15180969238ms, 45.16370275880554ms on average
accuracy: 97.79%; precision: 87.80%; recall: 90.28%; FB1: 89.02
token_eval micro F1: 0.9107763615295481

emb_class=elmo, enc_class=bilstm

train

* token_emb_dim in configs/config-elmo.json == 300 (ex, kor.glove.300k.300d.txt )
* elmo_emb_dim in configs/config-elmo.json == 1024 (ex, kor_elmo_2x4096_512_2048cnn_2xhighway_1000k* )
$ python preprocess.py --config=configs/config-elmo.json --data_dir=data/kmou2019 --embedding_path=embeddings/kor.glove.300k.300d.txt
$ python train.py --config=configs/config-elmo.json --save_path=pytorch-model-elmo-kor-kmou-morph.pt --data_dir=data/kmou2019 --elmo_options_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_options.json --elmo_weights_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_weights.hdf5 --use_crf

evaluation

$ python evaluate.py --config=configs/config-elmo.json --model_path=pytorch-model-elmo-kor-kmou-morph.pt --data_dir=data/kmou2019 --elmo_options_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_options.json --elmo_weights_file=embeddings/kor_elmo_2x4096_512_2048cnn_2xhighway_1000k_weights.hdf5 --use_crf
$ cd data/kmou2019; perl ../../etc/conlleval.pl < test.txt.pred ; cd ../..
$ cd data/kmou2019; python ../../etc/token_eval.py < test.txt.pred ; cd ../..
INFO:__main__:[F1] : 0.8803757522383678, 927
INFO:__main__:[Elapsed Time] : 126180ms, 135.92548596112312ms on average
accuracy: 97.30%; precision: 88.00%; recall: 88.08%; FB1: 88.04
token_eval micro F1: 0.8880736809241336
* --batch_size=64 --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.8809558502112778, 927
INFO:__main__:[Elapsed Time] : 927 examples, 118820ms, 127.98920086393089ms on average
accuracy: 97.29%; precision: 87.42%; recall: 88.78%; FB1: 88.10
token_eval micro F1: 0.8871589332511176
* --embedding_trainable
INFO:__main__:[F1] : 0.8755125951962508, 927
INFO:__main__:[Elapsed Time] : 125665ms, 135.366090712743ms on average
accuracy: 97.33%; precision: 87.32%; recall: 87.78%; FB1: 87.55
token_eval micro F1: 0.8897515527950312
* --use_char_cnn --batch_size=64 --lr_decay_rate=0.9
INFO:__main__:[F1] : 0.8799648248571009, 927
INFO:__main__:[Elapsed Time] : 927 examples, 108596ms, 116.96544276457884ms on average
accuracy: 97.37%; precision: 87.83%; recall: 88.16%; FB1: 88.00
token_eval micro F1: 0.8920632495607668
* --batch_size=64 --lr_decay_rate=0.9 , modify model.py for disabling glove
INFO:__main__:[F1] : 0.8822326125073057, 927
INFO:__main__:[Elapsed Time] : 118884ms, 128.0291576673866ms on average
accuracy: 97.35%; precision: 87.79%; recall: 88.66%; FB1: 88.22
token_eval micro F1: 0.8905289052890529
* --use_char_cnn --batch_size=64 --lr_decay_rate=0.9 , modify model.py for disabling glove
INFO:__main__:[F1] : 0.8825072886297376, 927
INFO:__main__:[Elapsed Time] : 118407ms, 127.51403887688984ms on average
accuracy: 97.43%; precision: 87.61%; recall: 88.90%; FB1: 88.25
token_eval micro F1: 0.8926606215608773

Citation

@misc{ntagger,
author = {dsindex},
title = {ntagger},
year = {2020},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/dsindex/ntagger}},
}

References

Named Entity Recoginition
transformers_examples
macro and micro precision/recall/f1 score
RoBERTa
- RoBERTa GLUE task setting
BERT Miniatures
- search range of hyperparameters
  - batch sizes: 8, 16, 32, 64, 128
  - learning rates: 3e-4, 1e-4, 5e-5, 3e-5
scalar mixtures of BERT all layers
- ScalarMixWithDropout
- ScalarMix
Poor Man’s BERT: Smaller and Faster Transformer Models
- https://github.com/hsajjad/transformers/blob/master/examples/run_glue.py
(pytorch) advanced_tutorial

Name		Name	Last commit message	Last commit date
Latest commit History 370 Commits
configs		configs
data		data
etc		etc
.gitignore		.gitignore
README.md		README.md
dataset.py		dataset.py
early_stopping.py		early_stopping.py
evaluate.py		evaluate.py
model.py		model.py
preprocess.py		preprocess.py
progbar.py		progbar.py
requirements.txt		requirements.txt
tokenizer.py		tokenizer.py
train.py		train.py
util.py		util.py
util_bert.py		util_bert.py

Navigation Menu

Repository files navigation

Description

Requirements

Data

CoNLL 2003 (English)

from etagger

data/conll2003

data/conll2003_truecase

Naver NER 2019 (Korean)

from HanBert-NER

data/clova2019

data/clova2019_morph

data/clova2019_morph_space

there is no test set. therefore, set valid.txt as test.txt.

KMOU NER 2019 (Korean)

from KMOU NER

data/kmou2019

Korean pretrained model

Korean GloVe, BERT, ELECTRA, ELMo

CoNLL 2003 (English)

experiments summary

Naver NER 2019 (Korean)

experiments summary

clova2019(eoj-based)

clova2019_morph(morph-based)

clova2019_morph_space(morph-based + space as token)

KMOU NER 2019 (Korean)

experiments summary

Citation

References

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Description

Requirements

Data

CoNLL 2003 (English)

from etagger

data/conll2003

data/conll2003_truecase

Naver NER 2019 (Korean)

from HanBert-NER

data/clova2019

data/clova2019_morph

data/clova2019_morph_space

there is no test set. therefore, set valid.txt as test.txt.

KMOU NER 2019 (Korean)

from KMOU NER

data/kmou2019

Korean pretrained model

Korean GloVe, BERT, ELECTRA, ELMo

CoNLL 2003 (English)

experiments summary

Naver NER 2019 (Korean)

experiments summary

clova2019(eoj-based)

clova2019_morph(morph-based)

clova2019_morph_space(morph-based + space as token)

KMOU NER 2019 (Korean)

experiments summary

Citation

References

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages