Developer chat


(Kaspar Lund) #606

@stas do you know if we can instruct the GPU to stop keeping a mem cache so that we can see the timeline for mem allocations.

When i restart the PC and/or jupyter notebook then i can see a surge in GPU-mem when starting training of a language model. i think it is when backprop starts. This would be easier to pin down without the cache hiding the amount of used memory


(Stas Bekman) #607

do you know if we can instruct the GPU to stop keeping a mem cache so that we can see the timeline for mem allocations.

I don’t know, perhaps there is a way to compile pytorch w/ caching disabled? Ask at http://discuss.pytorch.org/ and report back your findings?

Until, then try to run torch.cuda.empty_cache() at strategic points.

And you might find this cell-by-cell gpu memory logger that I have just released useful: https://github.com/stas00/ipygpulogger - it’s totally new so I’m still tweaking the interface (and feedback is welcome!). But the main reason I mentioned it to you is that it runs empty_cache() automatically for you before and after each cell is run to measure the gpu memory usage correctly. (and gc.collect() but that can be turned off)

I have also just discovered this pytorch CUDA memory profiler, which perhaps can be useful to you. https://gist.github.com/dojoteef/26cd46f7cc38b38e6f443c5f62411aa3

I have just started a new thread GPU Optimizations Central - let’s have that discussion over there and use that thread for compiling all the knowledge we collectively discover.

When i restart the PC and/or jupyter notebook then i can see a surge in GPU-mem when starting training of a language model. i think it is when backprop starts.

It could be this too: https://github.com/stas00/ipygpulogger#framework-preloading - if it’s the first 0.5GB then it certainly is the case.


(Kaspar Lund) #608

thx
I have installed it and removed a ton statements to measure used memory from my notebook/.py files.
Good idea to preload pytorch:


#610

New: we can now directly export the Learner which avoids having to redefine the model at inference time (it’s saved with the data). You can check the inference tutorial for all the details, but you basically say learn.export() when you are ready, then learn = load_learner(path) when you want to load your inference learner.

Breaking change: the adult_sample dataset has been updated, you need to manually destroy the one you have to trigger a download.


(Dien Hoa TRUONG) #611

learn.summary() seems have a bug right ?

In the latest version of fast.ai I encounter.


(Stas Bekman) #613

@jeremy changed it not to print() the output but to return the data. And ipython/jupyter doesn’t interpret special raw characters, you have to run the data through print().

In [1]: y = "\n".join(["a","b"])

In [2]: y
Out[2]: 'a\nb'

In [3]: print(y)
a
b

So basically you now have to do this:

print(learn.summary())

The function could probably be smart and detect if it’s in an ipython shell or not and print() instead of returning data then, but perhaps it’d be an inconsistent behavior.

Currently, fastai has an inconsistent mix of some functions returning data, others printing.

My guess is that the change was done to better support the use of fastai outside of jupyter environment, where “unsolicited” printing is not a correct function behavior.

Of course, the other solution is to have a set of ipython wrappers that:

def summary_p(self): print(self.summary()) 

and then you use the wrapper:

learn.summary_p()

or something like that.


(Andrij David) #614

is SWA already implemented in fast.ai v1 ? I found nothing from the docs.


#615

Not yet, @wdhorton said he was working on it I believe.


(Andrij David) #616

Ok thank you. @wdhorton I am available if you need help.


#617

Just finished:

  • LanguageModelLoader (used behind the scenes by TextLMDataBunch) has now been replaced by LanguageModelPreLoader which isn’t a DataLoader but an intermediate between the dataset and a pytorch DataLoader. It’s a Dataset and a Callback at the same time, and is responsible for reading a portion of the stream created by all the texts concatenated.
  • Which means we can have pre-loader now that are Callback. The only events we can call are on_epoch_begin or on_epoch_end since the multiprocessing in pytorch DataLoader (with num_workers>=1) makes a copy of the underlying dataset that is only synchronized at the end of the iteration.

(Kaspar Lund) #618

this is really nice and memory usage is down. THX

Here is a small suggestion for def getitem__(self, k:int): inserting the blow line just before the comment “#Returning the right portion”. will allow users to provide the token id’s in a format that match the vocab . FX: np.uint16 for a vocab of size 64k.

    if concat.dtype != np.int64: concat = concat.astype(np.int64)

#619

Will add.
Also note I removed the varying bptt because it doesn’t add anything now that we shuffle the texts at each batch (tested on witkitext-2).


(Kaspar Lund) #620

agree i could not measure any difference using p_bppt nor my own uniform distribution

I believe there is a ±1 offset issue between batches in the new version.
I had so many problems making my own indexing of the jagged array work, that i created a test to generate jagged arrays with continous numbers but random layout. I feel confident when i can handle 10000 different layouts

Here is a result with the ± issue from running on the newest version of LanguageModelPreLoader i fastai dev: https://github.com/kasparlund/nlp/blob/master/test_languagemodelloader.ipynb

when i tried again it failed because start became greater than end in getitem.

I can create an issue if you agree that there is an issue


(Deena Blumenkrantz) #621

get_transforms() with default settings threw the error below for histology images

"RuntimeError: B should have at least 2 dimensions, but has 1 dimensions instead"

I’m using images from this Kaggle competition:

First I set all the arguments for get_transforms() to zero to get a baseline

tfms = get_transforms(do_flip=False, 
                      flip_vert=False, 
                      max_rotate=0., 
                      max_zoom=0., 
                      max_lighting=0., 
                      max_warp=0., 
                      p_affine=0., 
                      p_lighting=0.)

Everything, I did below worked:

data = (ImageItemList.from_df(df=df, path=path, cols='fpaths')
                     .random_split_by_pct(valid_pct=0.2, seed=10)
                     .label_from_df(cols='class_label')
                     .transform(tfms, size=49)
                     .databunch(bs=128))

data.show_batch(rows=3, figsize=(7,7), hide_axis=False)

learn = create_cnn(data, models.resnet34, metrics=[error_rate, accuracy])

learn.fit_one_cycle(6)

Then I tried using the default options for get_transforms() and got the error:

tfms = get_transforms()

data = (ImageItemList.from_df(df=df, path=path, cols='fpaths')
                     .random_split_by_pct(valid_pct=0.2, seed=10)
                     .label_from_df(cols='class_label')
                     .transform(tfms, size=49)
                     .databunch(bs=128))

data.show_batch(rows=3, figsize=(7,7), hide_axis=False)

"RuntimeError: B should have at least 2 dimensions, but has 1 dimensions instead"

Finally, I narrowed the cause of the problem down to max_warp() by manually entering all of the defaults and changing each to zero, one at a time:

tfms = get_transforms(do_flip=True, 
                      flip_vert=False, 
                      max_rotate=10., 
                      max_zoom=1.1, 
                      max_lighting=0.2, 
                      max_warp=0., 
                      p_affine=0.75, 
                      p_lighting=0.75)

Thought I’d share this in case anyone else ran into the same issue.

It is impressive that fastai knows that these images should not have max_warp applied! Is this a bug?

I wonder if this same error will be thrown when I want to look at 3D images of cells and organelles? … TBD

The full error text was:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-25-0a63e3fd5550> in <module>
----> 1 data.show_batch(rows=3, figsize=(7,7), hide_axis=False)

~/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/basic_data.py in show_batch(self, rows, ds_type, **kwargs)
    151     def show_batch(self, rows:int=5, ds_type:DatasetType=DatasetType.Train, **kwargs)->None:
    152         "Show a batch of data in `ds_type` on a few `rows`."
--> 153         x,y = self.one_batch(ds_type, True, True)
    154         if self.train_ds.x._square_show: rows = rows ** 2
    155         xs = [self.train_ds.x.reconstruct(grab_idx(x, i, self._batch_first)) for i in range(rows)]

~/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/basic_data.py in one_batch(self, ds_type, detach, denorm)
    134         w = self.num_workers
    135         self.num_workers = 0
--> 136         try:     x,y = next(iter(dl))
    137         finally: self.num_workers = w
    138         if detach: x,y = to_detach(x),to_detach(y)

~/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/basic_data.py in __iter__(self)
     68     def __iter__(self):
     69         "Process and returns items from `DataLoader`."
---> 70         for b in self.dl:
     71             y = b[1][0] if is_listy(b[1]) else b[1]
     72             yield self.proc_batch(b)

~/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    466                 self.reorder_dict[idx] = batch
    467                 continue
--> 468             return self._process_next_batch(batch)
    469 
    470     next = __next__  # Python 2 compatibility

~/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _process_next_batch(self, batch)
    487         self._put_indices()
    488         if isinstance(batch, _utils.ExceptionWrapper):
--> 489             raise batch.exc_type(batch.exc_msg)
    490         return batch
    491 

RuntimeError: Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/data_block.py", line 486, in __getitem__
    x = x.apply_tfms(self.tfms, **self.tfmargs)
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/vision/image.py", line 113, in apply_tfms
    else: x = tfm(x)
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/vision/image.py", line 498, in __call__
    return self.tfm(x, *args, **{**self.resolved, **kwargs}) if self.do_run else x
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/vision/image.py", line 445, in __call__
    if args: return self.calc(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/vision/image.py", line 450, in calc
    if self._wrap: return getattr(x, self._wrap)(self.func, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/vision/image.py", line 167, in coord
    self.flow = func(self.flow, *args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/vision/transform.py", line 227, in symmetric_warp
    return _perspective_warp(c, targ_pts, invert)
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/vision/transform.py", line 213, in _perspective_warp
    return _apply_perspective(c, _find_coeffs(_orig_pts, targ_pts))
  File "/home/ubuntu/anaconda3/envs/fastai/lib/python3.7/site-packages/fastai/vision/transform.py", line 194, in _find_coeffs
    return torch.gesv(B,A)[0][:,0]
RuntimeError: B should have at least 2 dimensions, but has 1 dimensions instead

Lesson 1 Discussion ✅
#622

It’s a bug with the new version of pytorch, it has been fixed in master I believe.


(Bobak Farzin) #623

split from Fastai v1 install issues thread

Trouble with tests/test_vision_data.py

Second problem, is that when I run the tests in the latest pull ($make test) I get an error related to pulling some data in using mnist = untar_data(URLs.COCO_TINY)

Looking at the ~/.fastai/data/ directory, it seems that sometimes the HTTP call fails to pull in a .tgz and that leads to the untar failing. If I delete the empty .tgz, then I (sometimes) get success the second time around. I believe this is due to a socket.timeout

My solution is to remove all the files in that directory rm -r ~/.fastai/data/* and then try again. After a few tries, it all seems to come in and I can run all the tests. I don’t know why the timeout is an issue and an intermittent one at that.

Show install information:

=== Software === 
python        : 3.7.1
fastai        : 1.0.40.dev0
fastprogress  : 0.1.18
torch         : 1.0.0
nvidia driver : 396.51
torch cuda    : 9.0.176 / is available
torch cudnn   : 7401 / is enabled

=== Hardware === 
nvidia gpus   : 2
torch devices : 2
  - gpu0      : 12194MB | TITAN Xp
  - gpu1      : 12196MB | TITAN Xp

=== Environment === 
platform      : Linux-4.15.0-32-generic-x86_64-with-debian-stretch-sid
distro        : Ubuntu 16.04 Xenial Xerus
conda env     : fai_v1_dev
python        : /home/farzin/anaconda3/envs/fai_v1_dev/bin/python
sys.path      : 
/home/farzin/anaconda3/envs/fai_v1_dev/lib/python37.zip
/home/farzin/anaconda3/envs/fai_v1_dev/lib/python3.7
/home/farzin/anaconda3/envs/fai_v1_dev/lib/python3.7/lib-dynload
/home/farzin/anaconda3/envs/fai_v1_dev/lib/python3.7/site-packages
/home/farzin/fast_ai/fastai-fork

Fri Jan 11 14:32:56 2019    
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.51                 Driver Version: 396.51                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN Xp            Off  | 00000000:03:00.0 Off |                  N/A |
| 30%   47C    P8    21W / 250W |     12MiB / 12194MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN Xp            Off  | 00000000:04:00.0  On |                  N/A |
| 23%   36C    P8    17W / 250W |    979MiB / 12196MiB |      3%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    1      1433      G   /usr/lib/xorg/Xorg                           661MiB |
|    1      2423      G   compiz                                       302MiB |
|    1     22300      G   /usr/lib/firefox/firefox                       3MiB |
+-----------------------------------------------------------------------------+


(Bobak Farzin) #624

I am willing to look deeper into this and try to fix/debug if anyone has ideas about where to look. I don’t quite know where to even start on this particular one.


(Stas Bekman) #625

Great, here is what needs to be done, if you can resolve those via PR that would be great. thank you.

  1. the error is not user-friendly error

…/…/anaconda3/envs/fai_v1_dev/lib/python3.7/tarfile.py:2304: ReadError

it needs to catch this error and report that the file failed to download fully. Do note that the git version now stores a simple checksum (not md5 as it’s too slow) for each dataset. So use that checksum to validate that file was fully downloaded.

but I think if solution (2) is implemented (1) will no longer be needed as it’ll include error handling in it. But the checksum part will still have to be a part of the solution.

  1. do a retry 5 times on failure and if that still failed, have the function raise an exception that prints the instructions on how to fix this manually, i.e.:
mkdir -p ~/.fastai/datasets
cd ~/.fastai/datasets
wget -c path/file.tgz
tar -xvzf file.tgz

and rerun the code/test which should now work.

So that the user could find another way to get the dataset.

wget -c will download as much as it can in a single connection and then continue retrying until it gets the file downloaded fully, not sure about curl equivalent, but it has one.


(Bobak Farzin) #626

I will give it a shot and come back with questions.


(Bobak Farzin) #627

I have tried two times to submit a PR with updated docs. The only file I have touched in both PRs is fastai/docs_src/core.ipynb

Both times, my version has errors which I initially was able to replicate, but can no longer get on my local system. I went through a lot of steps to “clean up” my conda install, then to get the pip dev install working correctly in a clean environment. That seems to have addressed the issues on my side.

When I pull from master and run the make test and the docs_src/run_tests.sh I get no errors (well, I do get these download errors, but I am working on that…)

How can I proceed to debug if I can’t get the errors locally? I still feel like a noob in all of this space, so any feedback or help greatly appreciated!