NLP Paper

Parameter counts of several recently released pretrained language models.

source: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

1. Deep Learning from scrach 2_sub study

모두의 연구소 풀잎스쿨 11기 밑바닥부터 더 딥하게 배워보자 딥러닝 서브스터디

매주 1회 진행 (20/5/28~7/10)

paper1 (05/28/20)

• Efficient Estimation of Word Representations in Vector Space (original word2vec paper)

paper2 (06/04/20)

• Distributed Representations of Words and Phrases and their Compositionality (negative sampling paper)

paper3 (06/11/20)

• Understanding LSTM Networks (blog post overview)
• The Unreasonable Effectiveness of Recurrent Neural Networks (blog post overview)

paper4 (06/18/20, cancel)

• Sequence to Sequence Learning with Neural Networks (original seq2seq NMT paper)

paper5 (06/25/20)

• Neural Machine Translation by Jointly Learning to Align and Translate (original seq2seq+attention paper)

paper6 in main study (06/30/20)

• Attention Is All You Need

paper7 in main study (07/07/20)

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

paper8 (07/10/20)

• Deep contextualized word representations(ELMO)

2. beyondBERT

모두의 연구소 풀잎스쿨 11.5기 beyondBERT

매주 1회 진행 (20/06/20~8/29)

week02 (06/20/20)

• The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives
• How multilingual is Multilingual BERT?

week03 (06/27/20)

• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

week04 (07/04/20)

• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• Data Augmentation using Pre-trained Transformer Models

week05 (07/11/20)

• Reformer: The Efficient Transformer
• Longformer: The Long-Document Transformer

week06 (07/18/20)

• Mask-Predict: Parallel Decoding of Conditional Masked Language Models
• Unsupervised Data Augmentation for Consistency Training -> Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

week07 (07/25/20)

• You Impress Me: Dialogue Generation via Mutual Persona Perception
• Recipes for building an open-domain chatbot

week08 (08/01/20)

• ToD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogues
• A Simple Language Model for Task-Oriented Dialogue

week09 (08/08/20)

• ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation
• FastBERT: a Self-distilling BERT with Adaptive Inference Time

week10 (08/22/20)

• PoWER-BERT: Accelerating BERT inference for Classification Tasks
• TinyBERT: Distilling BERT for Natural Language Understanding

week11 (08/29/20)

• GPT3: Language Models are Few-Shot Learners
• T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

3. Model Implematation (+ Code Review)

NLP paper reading 및 model implementation 스터디

매주 1회 (20/07/20~현재 진행중)

No	Model	Framework(code)	Paper	Author	Submission date
S1-1	Transformer	Tensorflow(tutorial)	Attention Is All You Need	Google	2017/6
S1-2	ELMO	AllenNLP(GitHub)	Deep contextualized word representations	AllenNLP	2018/2
S1-3	GPT	TensorFlow(github)	Improving Language Understanding with Unsupervised Learning	OpenAI(post)	2018/6
S1-4	BERT	TensorFlow(github)	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	Google	2018/10
S2-1	GPT2	TensorFlow(github)	Language Models are Unsupervised Multitask Learners	OpenAI(post)	2019/2
S2-2	MASS	(github)	MASS: Masked Sequence to Sequence Pre-training for Language Generation	Microsoft	2019/5
S2-3	XLNet		XLNet: Generalized Autoregressive Pretraining for Language Understanding	+Google	2019/6
S2-4	RoBERTa	(github)	RoBERTa: A Robustly Optimized BERT Pretraining Approach	Facebook	2019/7
S2-5	ALBERT		ALBERT: A Lite BERT for Self-supervised Learning of Language Representations		2019/9
S2-6	DistilBERT		DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter	Hugging Face	2019/10
S2-7	BART		BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension		2019/10
S2-8	ELECTRA		ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators		2020/3

* Transformer
- 구현 언어:
- 구현 레러펀스:
- code:

* ELMO
- 구현 언어:
- 구현 레러펀스:
- code:

* GPT
- 구현 언어:
- 구현 레러펀스:
- code:

* BERT
- 구현 언어:
- 구현 레러펀스:
- code:

* GPT2
- 구현 언어:
- 구현 레러펀스:
- code:

Season1: 2020.7~12

week1 (07/20/20)

• Study Planning

week2 (08/10/20)

• Transformer: architecture

week3 (08/21/20)

• Transformer: label smoothing/beam search

week4 (08/28/20)

• Transformer: trainning/multi-GPU/experiment

week5 (09/04/20)

• ELMo paper review

week6 (09/14/20)

• ELMo char-CNN layer

week7 (09/21/20) -> update할것

• model

week8 (09/28/20)

• model

week9 (10/05/20)

• model

week10 (10/12/20)

• model

week11 (10/19/20)

• model

week12 (10/26/20)

• model

week13 (11/02/20)

• model

week14 (11/09/20)

• model

week15 (11/16/20)

• model

week16 (11/23/20)

• model

week17 (11/30/20)

• model

week18 (12/07/20)

• model

week19 (12/14/20)

• BERT

week20 (12/21/20)

• BERT

---

Season2. 2021.1~

week21 (1/16/21)

• GPT2 paper discussion(1) (~2.2 Input Representation)

week22 (1/20/21)

• GPT2 paper discussion(2) (3. Experiments~)

week23 (1/28/21)

• GPT2 paper discussion(2) (3. Experiments~)

week24 (2/4/21)

• model

week25 (2/18/21)

• model

week26 (2/25/21)

• model

week27 (3/4/21)

• model

week28 (3/11/21)

• model

week29 (3/18/21)

• model

week20 (3/25/21)

• model