Grow BERT

This directory contains code for On the Transformer Growth for Progressive BERT Training. The proposed method CompoundGrow speeds up BERT pre-training by 73.6% and 82.2% for the base and large models respectively while achieving comparable performances. Code will be released for reproduction and future studies.

Install TF Model Garden, pip will install all models and dependencies automatically.

pip install tf-models-official