Anatoly Potapov: Pre-training Transformers with Catalyst

Data fest Online 2020 Catalyst Workshop track Since the NLP’s Imagenet moment (emergence of ELMO, BERT), language models are widely used as a backbone for a variety of supervised tasks (intent classification, named entity recognition, question answering), etc At Tinkoff, we have tens of millions of unlabelled customer conversation samples. In such a scenario, it is highly beneficial to pre-train on in-domain data and having a custom vocabulary. We transferred our pre-training pipeline to the Catalyst framework and reduced our codebase while keeping features like distributed training and fp16 training. In my presentation, I will tell you how to pre-train transformers at scale with the Catalyst framework without writing lots of “infrastructure“ code. Register and get access to the tracks: Join the community: