Developer chat

[sgugger]

Indeed, it’s this one that causes problems:
c:\users\sylvain\anaconda3\envs\fastai1\lib\subprocess.py(997)_execute_child()
Which doesn’t help us since it’s not the file not found, it’s where the file wasn’t found in code

[jeremy (Jeremy Howard)]

I moved some of the docs around so that everything except CONTRIBUTING.md is properly included in the docs site and organized in the sidebar and gets a TOC. (@stas I know you were a bit unsure about whether it’s OK for fastai library users to see this info, but I think it’s OK since it’s clear about who it’s for.)

[lesscomfortable (Francisco Ingham)]

I think your idea is better, however I don’t think checking for signatures and and signing if unsigned will take longer since usually there will be more unmodified notebooks than modified notebooks

[ashaw (Andrew)]

http://docs.fast.ai/gen_doc.sgen_notebooks.html should have the lates on how to update notebook documentation and the associated html

in the base directory, run something like:
python fastai/gen_doc/sgen_notebooks.py --update_html=True

[stas (Stas Bekman)]

conda and the different dependencies twists and mismatching package names proved to be a bit of a hell.

So I had to build our own torchvision conda package that depends on pytorch-nightly conda package…

Please help me test that it (1) installs and (2) actually works.

Currently only linux-64/python3.6 conda build is available, so please only try if that’s your setup.

First clear out your environment:

conda uninstall fastai pytorch-nightly pytorch torchvision
pip uninstall fastai pytorch-nightly pytorch torchvision

now:

conda install -c pytorch pytorch-nightly
conda install -c fastai/label/test torchvision=0.2.1=pyhe7f20fa_0

and finally:

conda install -c fastai/label/test fastai

Let me know if you successfully install it first and then if you can use it even if you’re using editable install - this is just to test. you can uninstall it right away.

There is the deadline so your help is crucial.

Thank you!

[sgugger]

Had a few commits today and vision docs is almost finished, only vision.learner is missing for now.

[jeremy (Jeremy Howard)]

@sgugger that reminds me - not a big deal but you may want to rename vision_learner and text_learner in the module dependencies diagram too.

[stas (Stas Bekman)]

Indeed, it’s this one that causes problems:
c:\users\sylvain\anaconda3\envs\fastai1\lib\subprocess.py(997)_execute_child()
Which doesn’t help us since it’s not the file not found, it’s where the file wasn’t found in code

I suppose you are replying to Jeremy’s suggestion, could you please run the modified code I pasted which should tell us what is it trying to find?

I don’t know what to make of those strings you pasted - lacking context.

Thank you.

[stas (Stas Bekman)]

I think your idea is better, however I don’t think checking for signatures and and signing if unsigned will take longer since usually there will be more unmodified notebooks than modified notebooks

Please go ahead and proceed with your suggestion, @lesscomfortable, we can always measure/tweak it later if we discover that it’s becoming a hurdle. Thank you for looking into it.

[stas (Stas Bekman)]

And btw, we have a fledgling test suite already. Please see:

http://docs.fast.ai/developers.html#test-suite

And you may start adding tests to it as you create new features.

Or if you have some time on your hands, please, write tests for the existing ones.

Note that some of the v0 tests can be ported to v1. https://github.com/fastai/fastai/tree/master/tests
I ported a few subtests from tests/test_core.py of v0, but there a lot more there to port.

Thank you.

[stas (Stas Bekman)]

Some of you might appreciate this new addition:

Github commit emails with full diff of changes

I’m still doing some tweaks, but it’s mostly fantastic! I’m so happy to be able to get diffs by email! Thank you, @jeremy, for supporting this idea. You can subscribe or unsubscribe at any time on your own accord here: https://groups.google.com/forum/#!forum/fastai-diff Make sure you use a descent email client, surely don’t read it on google groups or gmail, you need to have a client that can properly render fixed fonts and not use a variable fonts which look horrible with diffs. But google d…

[sgugger]

@sgugger that reminds me - not a big deal but you may want to rename vision_learner and text_learner in the module dependencies diagram too.

It’s done.

I suppose you are replying to Jeremy’s suggestion, could you please run the modified code I pasted which should tell us what is it trying to find?

Here the output of the modified script

Path tools
Executing: tools\trust-origin-git-config
Traceback (most recent call last):
  File "tools\run-after-git-clone", line 31, in <module>
    run_script(path/"trust-origin-git-config")
  File "tools\run-after-git-clone", line 18, in run_script
    result = subprocess.run(cmd.split(), shell=False, check=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "C:\Users\Sylvain\Anaconda3\envs\fastai1\lib\subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "C:\Users\Sylvain\Anaconda3\envs\fastai1\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "C:\Users\Sylvain\Anaconda3\envs\fastai1\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

[stas (Stas Bekman)]

Thank you, @sgugger. It is all correct.

I think we probably need to add `python’ to the command, no? Try:

- cmd = f"{script}"
+ cmd = f"python {script}"

[sgugger]

Yup, that works fine.

[stas (Stas Bekman)]

yay, thank you for testing, @sgugger . This is now fixed in git.

[nok]

Looking great! Will there be some description on how to generate a doc website as well?

[jeremy (Jeremy Howard)]

The doc website is built with jekyll from docs/, so just follow the usual jekyll docs for that bit. To create the docs/ folder from the notebooks in docs_src/ follow the instructions here:

http://docs.fast.ai/gen_doc.gen_notebooks.html

Theres a tools/docs_src/sgen_notebooks.py script that let’s you do the important stuff from the command line.

[lesscomfortable (Francisco Ingham)]

Hey @stas, just pushed my suggestion. However, the original script is not working for me (I can see some notebooks in docs_src as not trusted). Do you know why that might be the case?

[stas (Stas Bekman)]

@jeremy, can you please put what you shared into some place in the documentation?

Plus, this should include instructions on how to add/rename documents - i.e. how to link them, etc.

In particular, the current incarnation of developers.md should be renamed to something like release.md, and we need placeholders for other dev docs. So perhaps we need something like:

• docs/dev/build.md (move the last part from CONTRIBUTING.md)
• docs/dev/test.md (how to write/run tests)
• docs/dev/docs.md (this discussion)
• docs/dev/release.md (currently developers.md)

Other naming is totally fine with me, this is just something to start with. I’m thinking more along the lines of mirroring the more or less standard make targets, which we may have down the road:

• make
• make test
• make docs
• make release

Thanks.

[stas (Stas Bekman)]

I will have a look shortly.

It works just fine on my side. And your change sped it up by x3 on my slow setup! excellent! we might add an additional speed up with timestamps down the road.

Are you sure it’s installed as a hook? re-run:

tools/trust-doc-nbs-install-hook

or run directly to test:

tools/trust-doc-nbs

update: I have just added the last_modified < last_checked file timestamp check, so it’s much faster now when only a few nbs have changed. please make sure to re-run tools/trust-doc-nbs-install-hook after git pull once.