Human Protein Atlas Competition starter code

Part 1 (2019)
42

I just git pull to the latest fastai codebase. And it looks like ImageItemList::from_df not longer supports create_func argument. You can also check by using git show dbcdb785 vision/data.py.

43

Noting the external source has 3 channels only. Yellow is excluded.

Care also needs to be taken to make sure the 3 channel photo is normalized/processed to be similar to the 4 channel “images”.
I’m wondering what the best approach is to train the 4 channel model using 3 channel images. Perhaps adding 512x512 array = 0 as new channel is the simplest.

Any thoughts on best practices in this scenario?

44

Hi @wdhorton I started from scratch again and cloned the starter code repo and checked that I had your lastest commit with grep fa75d39314e3f1bb4b578866e6d8c90057869974 and this checked out fine.

But the resnet50 datablocks notebook does not have the src.test.x.create_func nor src.test.xopen statement you mention above. I have src.valid.x.create_func = open_4_channel and src.valid.x.open statements. I tried entering and running a src.test.x.create_fun but this generated an error message -

src_valid_x
src_valid_x.png821×599 25.9 KB

45

Also the statement src.add_test(test_fnames, label=‘0’); does not print anything at all. But your notebook shows it printing information on Labellists and the y MultiCategoryList and the x ImageItemList.

46

One more THANK from me!

Do TTA working ?
When i trying TTA, a have error -

/opt/conda/lib/python3.7/site-packages/fastai/vision/tta.py in _TTA(learn, beta, scale, ds_type, with_loss)
     36     if beta is None: return preds,avg_preds,y
     37     else:
---> 38         final_preds = preds*beta + avg_preds*(1-beta)
     39         if with_loss:
     40             return final_preds, y, calc_loss(final_preds, y, learn.loss_func)

TypeError: mul(): argument 'other' (position 1) must be Tensor, not DatasetType
47

You should use the resnet_basic_datablocks notebook for compatibility with the latest fastai library release

48

Yes, i do.
Everything working fine, i get 0.470 on LB, exept i cant do TTA.

49

@wdhorton Everything is working, thank you, I think my brain was not working well.

1 Like
50

I’m trying to get fastai v1 working as a kaggle kernel. I’ve got the dev versions installed OK. I’m stuck on the error RuntimeError: DataLoader worker (pid 110) is killed by signal: Bus error. I’m thinking the num_worker=0 might be in the wrong place, any ideas?

https://www.kaggle.com/dromosys/human-protein-fastai-v5

I’ve tried

Thanks

51

I was getting this error in Google Colab too.

At least in Google Colab adding num_workers=0 in the data_block did the trick.

Ps. I thought we couldn’t install any custom packages when using GPU’s ( this option is disabled when turning the GPU on ). That’s why I didn’t tried to do this but you managed to install it with pip so I got it wrong.

Ps2. It’s worth to ask this in another general thread.

52

Hello @Simmo

I’ve forked your Kaggle Notebook and got it working,

  1. I didn’t install the developer version. It seems they’ve upgraded the fastai version in Kaggle kernels. It was already installed version 1.0.29. Installing directly from the repo was working at all, so I just used the version installed
  2. You misspelled the parameter, it’s not num_worker but num_workers and also this parameter goes inside the databunch creator, along with the batchsize, this worked :
...databunch(bs=bs, num_workers=1)

So you weren’t really using the smaller batch size and the wrong parameter made no effect, it just got ignored instead of giving an error.

I’ve struggled with this myself for the past few weeks, I’m guessing the use of **kwargs in the function definitions prevent errors by wrong parameters since it can receive anything.

Cheers

53

Thanks got past that error too. now hit the next problem. Seems the update has stopped cuda from working.

54

Hello Simmo,

It was taking forever to run the code, it turns out it wasn’t using the GPU at all.

The message in the board on the left is “Custom Packages are not support in GPU instances”.

After installing torch 1.0 and fastai 1.0, the code works but don’t use it the GPU.

You can check it by running torch.cuda.is_available() and it returns False.

That explains why there is no kernel in fastai 1.0 already.

55

Why are you setting num_workers=1? Won’t this reduce the speed?

56

Yes, but this only applies to the docker container in Kaggle and sometimes Colab, but I’m not having any more problem in Colab. The following error is common in Kaggle Kernels RuntimeError: DataLoader worker (pid 173) is killed by signal when using multiple cores and your model uses memory closed to the limit. If multiprocessing is used the default shared memory of the docker container isn’t enough.

Disregard these in other instances and use the cores at your disposal.

More info: https://www.kaggle.com/product-feedback/72606

1 Like