The youth climate movement is on the news everywhere currently (at least in Europe). Some researchers in the DL community also start thinking about the impact on energy consumption and CO2 emissions that this field has.
I think this paper published 2 days ago makes some very interesting and important points that are worth thinking about:
1 - The energy and climate implications of deep learning training.
While the paper uses the newer NLP architectures and huge amounts of data as examples, the implications remain even when training smaller models. (Just think of the thousands of people on kaggle and the amount of GPU power used)
I am not sure that the measurement methodology is actually very precise, but even taking them as ballpark estimates shows the importance of being more aware of the impact and the need to strive for higher efficiency.
2 - The implications of making NLP research too expensive for academics.
It has become so resource intensive in terms of compute (= cost) that academics cannot “compete” anymore with corporate research. The authors advocate publicly financed compute cloud for academics as a possible mitigation strategy.
Recent progress in hardware and methodology for training neural networks has ushered in a new generation of large networks trained on abundant data. These models have obtained notable gains in accuracy across many NLP tasks. However, these accuracy improvements depend on the availability of exceptionally large computational resources that necessitate similarly substantial energy consumption. As a result these models are costly to train and develop, both financially, due to the cost of hardware and electricity or cloud compute time, and environmentally, due to the carbon footprint required to fuel modern tensor processing hardware. In this paper we bring this issue to the attention of NLP researchers by quantifying the approximate financial and environmental costs of training a variety of recently successful neural network models for NLP. Based on these findings, we propose actionable recommendations to reduce costs and improve equity in NLP research and practice.
This post is just to start up some discussion, what is your take on this?