Fine-tuning large neural language models for biomedical natural language processing.

Abstract

Large neural language models have transformed modern natural language processing (NLP) applications. However, fine-tuning such models for specific tasks remains challenging as model size increases, especially with small labeled datasets, which are common in biomedical NLP. We conduct a systematic study on fine-tuning stability in biomedical NLP. We show that fine-tuning performance may be sensitive to pretraining settings and conduct an exploration of techniques for addressing fine-tuning instability. We show that these techniques can substantially improve fine-tuning performance for low-resource biomedical NLP applications. Specifically, freezing lower layers is helpful for standard BERT- <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>B</mi> <mi>A</mi> <mi>S</mi> <mi>E</mi></mrow> </math> models, while layerwise decay is more effective for BERT- <math xmlns="http://www.w3.org/1998/Math/MathML"><mrow><mi>L</mi> <mi>A</mi> <mi>R</mi> <mi>G</mi> <mi>E</mi></mrow> </math> and ELECTRA models. For low-resource text similarity tasks, such as BIOSSES, reinitializing the top layers is the optimal strategy. Overall, domain-specific vocabulary and pretraining facilitate robust models for fine-tuning. Based on these findings, we establish a new state of the art on a wide range of biomedical NLP applications.