Developer chat

jeremy · September 29, 2018, 6:12pm

Working on it:

http://docs.fast.ai/gen_doc.html

stas · September 29, 2018, 7:14pm

Hmm, but now we no longer have .md to work with. there is no way to preserve the easy to edit format?

we have developers.md at the moment, we can’t have the same for doc gen and everything else?

Also most of the instructions that got moved from CONTRIBUTING aren’t belonging to doc-gen, should be in a different file.

stas · September 29, 2018, 7:39pm

heads up to a few renames under tools/:

git mv tools/trust-doc-nbs tools/trust-doc-nbs-install-hook
git mv tools/trust-doc-nbs-hook tools/trust-doc-nbs

but it should be transparent to you if you use tools/run-after-git-clone.

and another rename:

git mv tools/build tools/sync-nb-exports

so now it’s easier to know what we are trying to do.

jeremy · September 30, 2018, 5:21am

We’re using our documentation framework to document our documentation framework.

sgugger · September 30, 2018, 4:52pm

In case you missed it, fastai_pytorch is now merged on the fastai main repo. PRs should now be done there and only then. The old fastai library is in the old folder, the docs sources have moved in this repo.

yashuseth · September 30, 2018, 7:45pm

All the Source links (that point to the source code) in the documentation results in a 404. Could be due to these recent changes (fastai_pytorch is now merged on the fastai main repo).

Kaspar · October 1, 2018, 5:25am

So congrates with the major upgrade of fastai.

Will you incorporate the the brilliant notebooks like 001a_nn_basics.ipynb later or will they stay in “fastai old”

jeremy · October 1, 2018, 5:35am

001a will become an official pytorch tutorial on their site. The remaining notebooks will form the basis of the part2 course next year.

Kaspar · October 1, 2018, 9:24am

I believe the adresses on https://anaconda.org/fastai/fastai are point on the old repos ?

jeremy · October 1, 2018, 4:44pm

Thanks for the reminder

stas · October 1, 2018, 4:52pm

Thank you, @Kaspar, will be fixed in the next upload.

update: done

sgugger · October 1, 2018, 4:57pm

I think they’re all back working now. Thanks for pointing this out.

TheShadow29 · October 1, 2018, 8:49pm

EDIT: I solved it. I had to first run conda install conda. After that it worked flawlessly.

Leaving the rest of it verbatim, if someone also runs into the same error.

I am not sure why, but I am unable to install fastai on a new environment. I am trying conda install -c pytorch -c fastai fastai pytorch-nightly cuda90 (since I have cuda9.0). However, I am getting the following error:

Output I am getting:
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

fastai
fastprogress

Current channels:

To search for alternate channels that may provide the conda package you’re
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

I was able to install the fastai_v1 repository earlier via pip install -e .

Can anyone confirm the error? the anaconda repo seems correctly linked, so not sure why there is an error.

stas · October 2, 2018, 12:36am

Thank you for the report, @TheShadow29.

Can you please elaborate so that we know whether this is something that needs to be documented? How could you possibly have conda environment w/o conda installed? I’m not sure how you could run conda to install conda. Did you perhaps mean conda update conda?

Hmm, I have just tried conda uninstall conda and it actually allows that. I didn’t proceed, I am not sure why this is even possible. There must be a default thin conda util that gets installed by default. Too bad you didn’t catch --version before installing conda.

I think the reason the unupdated conda wasn’t finding the packages, is because -c fastai is not really a channel, but just a user on anaconda.org, they might have made it possible to use user’s account as a channel somewhat recently (it is still not fully supported by conda-build).

elmarculino · October 2, 2018, 1:07am

I’m having a hard time creating a fastai_v1 enviroment. Here is what I’m doing:

conda create --name fastai_v1
conda activate fastai_v1
conda install -c pytorch -c fastai fastai pytorch-nightly cuda92
conda uninstall fastai
pip install -e . ##inside the fastai repository folder

After that, when I try to run a notebook of the examples folder, a get this error:

ModuleNotFoundError: No module named 'torch'

I can import the torch package in the python terminal:

>>> import torch; torch.__version__
'1.0.0.dev20180930'

Should I do something different?

What is the environment.yml file in the fastai root folder for in this new version?

stas · October 2, 2018, 2:35am

heads up: please don’t make ‘import fastai’ rely on any of the jupyter stuff, as it breaks conda-build - none of the jupyter stuff is on conda - it be a monumental project to put all the dependencies there.
So I had to remove this:

fastai/__init__.py:
- from .gen_doc.nbdoc import nbshow

stas · October 2, 2018, 2:54am

I’m trying to get the environment tested w/o needing to run the notebook.

This worked:

jupyter nbconvert --execute --ExecutePreprocessor.timeout=600 --to notebook examples/tabular.ipynb

This fails:

jupyter nbconvert --execute --ExecutePreprocessor.timeout=600 --to notebook examples/cifar.ipynb

[NbConvertApp] WARNING | Config option `template_path` not recognized by `NotebookExporter`.
[NbConvertApp] Converting notebook examples/cifar.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: python3
[NbConvertApp] ERROR | Error while converting 'examples/cifar.ipynb'
Traceback (most recent call last):
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 393, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 174, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 192, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/exporters/notebook.py", line 31, in from_notebook_node
    nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 134, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 311, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 262, in preprocess
    nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/home/stas/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 286, in preprocess_cell
    raise CellExecutionError.from_cell_and_msg(cell, out)
nbconvert.preprocessors.execute.CellExecutionError: An error occurred while executing the following cell:
------------------
untar_data(CIFAR_PATH)
------------------

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-2-aac8ec0ed7ac> in <module>()
----> 1 untar_data(CIFAR_PATH)

/mnt/disc1/fast.ai-1/br/fastai/master/fastai/docs.py in untar_data(path)
     25
     26 def untar_data(path):
---> 27     download_data(path.name)
     28     if not path.exists(): tarfile.open(f_name(path), 'r:gz').extractall(DATA_PATH)
     29

/mnt/disc1/fast.ai-1/br/fastai/master/fastai/docs.py in download_data(name)
     22 def download_data(name):
     23     dest = DATA_PATH/f_name(name)
---> 24     if not dest.exists(): download_url(f'{URL}{f_name(name)}', dest)
     25
     26 def untar_data(path):

/mnt/disc1/fast.ai-1/br/fastai/master/fastai/core.py in download_url(url, dest, overwrite)
    153
    154     with open(dest,'wb') as f:
--> 155         pbar = progress_bar(range(file_size), auto_update=False)
    156         nbytes,buffer = 0,[1]
    157         while len(buffer):

TypeError: __init__() got an unexpected keyword argument 'auto_update'
TypeError: __init__() got an unexpected keyword argument 'auto_update'

update: it breaks in jupyter too, the same way.

It’d probably make a good basic test to just successfully run the example notebooks from the command line.

jeremy · October 2, 2018, 3:45am

That’s an out of date fastprogress version.

stas · October 2, 2018, 4:00am

@elmarculino, I have just followed your steps plus:

pip install jupyter_contrib_nbextensions ipywidgets

and loading pytorch works for me from the jupyter notebook.

Perhaps print your sys.path and see whether you’re using the wrong environment when you run jupyter notebook?

elmarculino · October 2, 2018, 4:40am

You are right. The path was wrong.

From notebook, using jupyter lab, i got the error ModuleNotFoundError: No module named ‘torch’:

['',
 '/home/marco/anaconda3/lib/python36.zip',
 '/home/marco/anaconda3/lib/python3.6',
 '/home/marco/anaconda3/lib/python3.6/lib-dynload',
 '/home/marco/anaconda3/lib/python3.6/site-packages',
 '/mnt/ssd32gb/Arquivos/MachineLearning/fastai_pytorch',
 '/home/marco/anaconda3/lib/python3.6/site-packages/IPython/extensions',
 '/home/marco/.ipython']

From notebook, using jupyter notebook:

['',
 '/mnt/ssd32gb/Arquivos/MachineLearning/fastai-repo/examples',
 '/home/marco/anaconda3/envs/fastai_v1/lib/python37.zip',
 '/home/marco/anaconda3/envs/fastai_v1/lib/python3.7',
 '/home/marco/anaconda3/envs/fastai_v1/lib/python3.7/lib-dynload',
 '/home/marco/anaconda3/envs/fastai_v1/lib/python3.7/site-packages',
 '/mnt/ssd32gb/Arquivos/MachineLearning/fastai-repo',
 '/home/marco/anaconda3/envs/fastai_v1/lib/python3.7/site-packages/IPython/extensions',
 '/home/marco/.ipython']

Here the kernel always restarts:

[W 01:19:16.595 NotebookApp] Notebook fastai-repo/examples/cifar.ipynb is not trusted
[I 01:19:17.113 NotebookApp] Kernel started: ed9289cd-5b3c-4acc-86fd-5e702248e09e
[I 01:19:17.681 NotebookApp] Adapting to protocol v5.1 for kernel ed9289cd-5b3c-4acc-86fd-5e702248e09e
[I 01:19:32.109 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
WARNING:root:kernel ed9289cd-5b3c-4acc-86fd-5e702248e09e restarted
[I 01:20:05.127 NotebookApp] KernelRestarter: restarting kernel (1/5), keep random ports
WARNING:root:kernel ed9289cd-5b3c-4acc-86fd-5e702248e09e restarted

Python console:

>>> sys.path
['', 
'/home/marco/anaconda3/envs/fastai_v1/lib/python37.zip', 
'/home/marco/anaconda3/envs/fastai_v1/lib/python3.7', 
'/home/marco/anaconda3/envs/fastai_v1/lib/python3.7/lib-dynload', 
'/home/marco/anaconda3/envs/fastai_v1/lib/python3.7/site-packages', 
'/mnt/ssd32gb/Arquivos/MachineLearning/fastai-repo']

I will reinstall my anaconda and try again. Thanks for your help!