bert [b23ddbd4]

Tags: nlp

• huggingface.co/transformers/bertology.html\
• github.com/huggingface/transformers/blob/master/examples/research_projects/bertology/run_bertology.py

• attention vs self-attention

  • attention is an operator that lets you collapse a sequence of vectors into a single vector

  • self attention is used when the vectors you're attending over were produced by a nn with the same params and the same input sequence

1. distilled bert

• arxiv.org/abs/1910.01108

2. Exploring bert

• huggingface.co/exbert/?model=distilgpt2&modelKind=bidirectional&sentence=The%20girl%20ran%20to%20a%20local%20pub%20to%20escape%20the%20din%20of%20her%20city.&layer=0&heads=..0,1,2,3,4,5,6,7,8,9,10,11&threshold=0.7&tokenInd=null&tokenSide=null&maskInds=..&hideClsSep=true