Transformer_Model

This is full Transformer model base Pytorch.

Contents

• Model Architecture
  • Encoder and Decoder Stacks
    • Encoder
    • Decoder
    • Attention
    • Applications of Attention in our Model
  • Position-wise Feed-Forward Networks
  • Embeddings and Softmax
  • Positional Encoding
  • Full Model
• Training
  • Batches and Masking
  • Training Loop
  • Training Data and Batching
  • Hardware and Schedule
  • Optimizer
  • Regularization
    • Label Smoothing

How to run

python MakeModel.py

Reference & Learn more

http://nlp.seas.harvard.edu/2018/04/03/attention.html