PyCharm IDE debugger exception

fastai fastai dev
1

Usually, I use PyCharm IDE and its debugger to develop Python projects. However, when I am trying to debug fastai’s training loop, I am getting an exception:

Connected to pydev debugger (build 182.4323.49)
epoch  train_loss  valid_loss  accuracy  error_rate
Traceback (most recent call last):
  File "/home/ck/.pycharm_helpers/pydev/pydevd.py", line 1664, in <module>
    main()
  File "/home/ck/.pycharm_helpers/pydev/pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/home/ck/.pycharm_helpers/pydev/pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/home/ck/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/ck/code/open_source/fastai_logger/logger.py", line 54, in <module>
    main()
  File "/home/ck/code/open_source/fastai_logger/logger.py", line 48, in main
    learn.fit(3, callbacks=[cb])
  File "/home/ck/code/fastai_v1/repo/fastai/basic_train.py", line 162, in fit
    callbacks=self.callbacks+callbacks)
  File "/home/ck/code/fastai_v1/repo/fastai/basic_train.py", line 82, in fit
    for xb,yb in progress_bar(data.train_dl, parent=pbar):
  File "/home/ck/anaconda3/envs/fastai/lib/python3.7/site-packages/fastprogress/fastprogress.py", line 65, in __iter__
    for i,o in enumerate(self._gen):
  File "/home/ck/code/fastai_v1/repo/fastai/basic_data.py", line 74, in __iter__
    for b in self.dl: yield self.proc_batch(b)
  File "/home/ck/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 626, in __next__
    self._shutdown_workers()
  File "/home/ck/anaconda3/envs/fastai/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 713, in _shutdown_workers
    w.join()
  File "/home/ck/anaconda3/envs/fastai/lib/python3.7/multiprocessing/process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "/home/ck/anaconda3/envs/fastai/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/home/ck/anaconda3/envs/fastai/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

I am not sure what is the reason, because I have never had this issue with other projects. I am able to debug remote code, Vagrant machines, Docker containers, etc. but in this specific case, the code fails.

Probably the problem is related to built-in multiprocessing library or something, but PyCharm allows you to debug multiprocessing (multithreading) apps as well.

I wonder, do anybody among fastai developers and contributors using this IDE? Otherwise, which tools do you prefer? Do you use some kind of text editor (Atom, VSCode) + console debugger?

2

Personally I use jupyter notebooks to test new functionalities (like you did for the CSVCallback) and I find it’s the most useful way to easily interact/debug/refactor.

When putting into modules, I like VSCode a lot, but by this time I only have to check I don’t break anything when copy-pasting so running pytest and the examples/lessons is enough to debug.

I don’t know pycharm and I have no idea why it throws this error.

3

I ran into similar problems while debugging with vscode. The vscode python experimental debugger became very slow once fastai training started, and the debugger kept resetting itself during a debugging session. Other vscode users were complaining about similar issues when dealing with many threads.

I finally found the solution an hour ago, by tracing the code to figure out what was generating the second thread. (I had already disabled num_workers everywhere since it doesn’t offer performance advantages in my domain.) It turns out it’s tqdm that spawns a thread by default when it’s used the first time. My solution was to do this at the top of my imports:

from tqdm import tqdm        
# Disable tqdm thread, that causes huge slowdowns with vscode experimental python debugger. 
# See https://pypi.org/project/tqdm/
# Must be set before any tqdm function is called.
tqdm.monitor_interval = 0

Now debugging is very fast, all the time. And it’s way less painful than debugging in a cell in Jupyter, especially when dealing with complex data structures like tracking pytorch’s backward propagation.

I hope this helps,

Christian

4

Yes, agree, Jupyter notebooks are great. The main reason I like PyCharm is their visual debugger and linting capabilities :smile: Very handy for huge projects.

Hm, interesting, maybe my problem is also somehow related to progress bars rendering. I’ll make a try to disable them.

I guess that at the end of the day I’ll create an issue on JetBrains support forum.

5

This seems to be a Python 3.7 specific issue.

I was able to avoid this and run the debugger by:

  1. Creating a new Anaconda environment with Python 3.6, Pytorch 1.0, and fastai
  2. pip install dataclasses

Everything works smoothly now!

2 Likes
6

Hm, that’s interesting. I have switched to the good old breakpoints and Jupyter/Vim :smile: However, it is great to know how to solve this thing for PyCharm!