Converting a fp16 model to fp32 for a GAN?

GrahamAcademy · June 1, 2020, 7:29pm

It looks like the Fastai GAN leaner does not support fp16. Is it possible to first train the generator and critic at fp16 to speed up training and then convert them to fp32 for the GAN portion of the training? I feel like this could really speed everything up.

I feel like this shouldn’t be impossible yet with all of my google skills I can’t seem to find an answer.

GrahamAcademy · June 17, 2020, 5:31pm

model = convert_network(model, dtype=torch.float32)

I found this searching around on the webs. Would this be able to convert a model to fp32 so it can be trained in a GAN in fastai? I found it in the fastai fp16 callback.

github.com

fastai/fastai2/blob/master/fastai2/callback/fp16.py

# AUTOGENERATED! DO NOT EDIT! File to edit: nbs/18_callback.fp16.ipynb (unless otherwise specified).

__all__ = ['get_master', 'to_master_grads', 'to_model_params', 'test_overflow', 'grad_overflow', 'copy_clone',
           'ModelToHalf', 'MixedPrecision', 'mixed_precision_one_batch', 'NativeMixedPrecision']

# Cell
from ..basics import *
from .progress import *

# Cell
from ..fp16_utils import convert_network, model_grads_to_master_grads, master_params_to_model_params

# Cell
from torch.nn.utils import parameters_to_vector

def get_master(opt, flat_master=False):
    model_params = [[param for param in pg if param.requires_grad] for pg in opt.param_lists]
    if flat_master:
        master_params = []
        for pg in model_params:

This file has been truncated. show original

ilovescience · June 18, 2020, 2:57am

You should be able to just do learn.to_fp32()

GrahamAcademy · June 18, 2020, 4:39pm

Oh darn really? I will give it a shot. So I will load the fp16 models and then do to_fp32 on them before loading them in to the GAN for training.

ilovescience · June 18, 2020, 5:34pm

Yes this should work…