Awesome tool! I’ll look into the missing anchors and why some of the links are broken
fantastic.
Here is one for docs-dev:
Access to these URLs denied by robots.txt, so we couldn't check them:
- https://test.pypi.org/account/register/
- https://www.sciencedirect.com/science/article/pii/S0167642309000343
http://docs-dev.fast.ai/release
- (408:118) 'backport..' => http://docs-dev.fast.ai/release#backporting_release_branch_to_master (HTTP 200 but missing anchor)
- (415:85) 'Detailed..' => http://docs-dev.fast.ai/release#Detailed_Release_Process (HTTP 200 but missing anchor)
- (513:122) 'backport..' => http://docs-dev.fast.ai/release#backporting_release_branch_to_master (HTTP 200 but missing anchor)
- (1335:61) 'Runs' => https://dev.azure.com/fastdotai/fastai/_TestManagement/Runs?runId=1&_a=runCharts (HTTP 302 => 203)
- redirect path:
- https://dev.azure.com/fastdotai/fastai/_TestManagement/Runs?runId=1&_a=runCharts (302)
- https://app.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2Ffastdotai%2Ffastai%2F_TestManagement%2FRuns%3FrunId%3D1%26_a%3DrunCharts&redirect=1&aadinmsa=true&hid=4cefdb46-0d1b-46ae-8aac-3fa9b73a5243&context=eyJodCI6MiwiaGlkIjoiZWMxODk5YTUtNTZkNC00ODA5LTljYzQtZDg3MGRiZjYwZDdiIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSIsImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSJdfQ2 (203)
- (1340:6) 'Notifica..' => https://dev.azure.com/fastdotai/fastai/_settings/notifications (HTTP 302 => 203)
- redirect path:
- https://dev.azure.com/fastdotai/fastai/_settings/notifications (302)
- https://app.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2Ffastdotai%2Ffastai%2F_settings%2Fnotifications&redirect=1&aadinmsa=true&hid=4cefdb46-0d1b-46ae-8aac-3fa9b73a5243&context=eyJodCI6MiwiaGlkIjoiZWMxODk5YTUtNTZkNC00ODA5LTljYzQtZDg3MGRiZjYwZDdiIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSIsImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSJdfQ2 (203)
http://docs-dev.fast.ai/release.html
- (408:118) 'backport..' => http://docs-dev.fast.ai/release.html#backporting_release_branch_to_master (HTTP 200 but missing anchor)
- (415:85) 'Detailed..' => http://docs-dev.fast.ai/release.html#Detailed_Release_Process (HTTP 200 but missing anchor)
- (513:122) 'backport..' => http://docs-dev.fast.ai/release.html#backporting_release_branch_to_master (HTTP 200 but missing anchor)
- (1335:61) 'Runs' => https://dev.azure.com/fastdotai/fastai/_TestManagement/Runs?runId=1&_a=runCharts (HTTP 302 => 203)
- redirect path:
- https://dev.azure.com/fastdotai/fastai/_TestManagement/Runs?runId=1&_a=runCharts (302)
- https://app.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2Ffastdotai%2Ffastai%2F_TestManagement%2FRuns%3FrunId%3D1%26_a%3DrunCharts&redirect=1&aadinmsa=true&hid=4cefdb46-0d1b-46ae-8aac-3fa9b73a5243&context=eyJodCI6MiwiaGlkIjoiZWMxODk5YTUtNTZkNC00ODA5LTljYzQtZDg3MGRiZjYwZDdiIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSIsImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSJdfQ2 (203)
- (1340:6) 'Notifica..' => https://dev.azure.com/fastdotai/fastai/_settings/notifications (HTTP 302 => 203)
- redirect path:
- https://dev.azure.com/fastdotai/fastai/_settings/notifications (302)
- https://app.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2Ffastdotai%2Ffastai%2F_settings%2Fnotifications&redirect=1&aadinmsa=true&hid=4cefdb46-0d1b-46ae-8aac-3fa9b73a5243&context=eyJodCI6MiwiaGlkIjoiZWMxODk5YTUtNTZkNC00ODA5LTljYzQtZDg3MGRiZjYwZDdiIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSIsImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSJdfQ2 (203)
http://docs-dev.fast.ai/style
- (336:189) 'APL' => https://en.wikipedia.org/wiki/APL_%5C(programming_language%5C) (HTTP 404)
- (336:268) 'J' => https://en.wikipedia.org/wiki/J_%5C(programming_language%5C) (HTTP 404)
- (336:343) 'K' => https://en.wikipedia.org/wiki/K_%5C(programming_language%5C) (HTTP 404)
- (347:63) 'Iverson’s' => https://en.wikipedia.org/wiki/Kenneth_E._Iverson (connection failed)
- (357:91) 'Arthur W..' => https://en.wikipedia.org/wiki/Arthur_Whitney_%5C(computer_scientist%5C) (HTTP 404)
http://docs-dev.fast.ai/style.html
- (336:189) 'APL' => https://en.wikipedia.org/wiki/APL_%5C(programming_language%5C) (HTTP 404)
- (336:268) 'J' => https://en.wikipedia.org/wiki/J_%5C(programming_language%5C) (HTTP 404)
- (336:343) 'K' => https://en.wikipedia.org/wiki/K_%5C(programming_language%5C) (HTTP 404)
- (347:63) 'Iverson’s' => https://en.wikipedia.org/wiki/Kenneth_E._Iverson (connection failed)
- (357:91) 'Arthur W..' => https://en.wikipedia.org/wiki/Arthur_Whitney_%5C(computer_scientist%5C) (HTTP 404)
Errors. Checked 742 links, 77 destination URLs (2 ignored), 7 have errors, 6 have warnings, 4 have info.
Update:
I fixed docs-dev links, beside a strange false negative:
http://docs-dev.fast.ai/style.html
- (336:189) 'APL' => https://en.wikipedia.org/wiki/APL_(programming_language) (connection failed)
- (347:63) 'Iverson’s' => https://en.wikipedia.org/wiki/Kenneth_E._Iverson (connection failed)In order for it not to be interfering (as there are so many broken anchors and some urls), I have it running as a cron-job for now at 3am every day for now.
You can see the outputs here:
https://dev.azure.com/fastdotai/fastai/_build?definitionId=5
Once all the broken links/anchors are fixed, I will then move it into a normal CI state, so that any future commits that break links will get detected right away.
Andew, I think you should set up an indentical linkcheck on your setup. See: https://pub.dartlang.org/packages/linkcheck for steps (you need to install dart, and then linkcheck). If you’re on linux you can copy the CI script.
Is it possible to fix http://docs-dev.fast.ai/ http://docs.fast.ai/ so that when a user clicks on an entry at TOC (or any internal link), the url in the location bar changes accordingly? Currently it’s very confusing and makes it difficult to share correct links.
to see what’s I’m talking about, go to say:
http://docs-dev.fast.ai/release.html#step-by-step-release-process
and click on " Detailed Release Process" in the first paragraph. The url remains unchanged, but it should become:
http://docs-dev.fast.ai/release.html#detailed-release-process
The problem is that if someone is now reading this section and they want the paste somewhere a direct link to it, a lot of times they will link to the wrong section.
If I click on the link icon next to the item it then changes to the correct url, but I don’t understand why is the strange behavior. A lot of people will just copy the contents of their location bar, without paying attention, since that’s how the web normally works.
I hope you can reproduce what I’m talking about.
I’m sure it’s possible, but we’re using an external TOC script so none of us know how. We can look at it after the course if no javascript gurus in the course find the time to do so.
Another issue - the sites use both w/ and w/ .html links at the same time:
http://docs-dev.fast.ai/style
http://docs-dev.fast.ai/style.html
perhaps we could stick to one style?
This is important for helping with link errors detection, currently linkchecker spews out double the error reports because of that.
while searching for a better nvidia-smi parser, I stumbled upon:
PyCUDA lets you access Nvidia’s CUDA parallel computation API from Python.
The neat thing about it is that we can access GPU data (like memory usage) w/o needing nvidia-smi. Someone posted:
I’m not sure why pytorch doesn’t provide a full access to NVIDIA memory management API. Or perhaps I just couldn’t find it. All I found is partial memory management.
currently 2018-ed pycuda is on pypi, and conda has an older 2017 version in a private channel. I guess we can’t rely on it due to no official releases on conda.
1 Like
If you use bash-git-prompt, I have just tweaked the most used version of it to handle the display of user/repo. I submitted a PR, don’t know whether it’ll be accepted or not, but you can use my patched version if you want to.:
(active-env) path [gituser/repo:branch|status]>
Now I get prompts like:
(pytorch-dev) /fastai/ci-experiments [fastai/fastai:ci-experiments|·6]>
(pytorch-dev) /fastai/linkcheck [fastai/fastai_docs:master]>
(pytorch-dev) /stas00/fork [stas00/fastai:master|·3]>
(pytorch-dev) /fastai/wip [fastai/fastai:master|+2?10·3]>
The numbers after the branch are modified/untracked/stashed counts. The leading (pytorch-dev) is the currently activated conda env name.
Without that setup I have to use precise directories and place forked versions in a different place from original versions, otherwise it’s too easy to get confused and commit to the original repo when I intend to commit to the fork.
Of course this only useful if you have commit access to the original repo and the fork.
But it’s a very handy tool regardless since it gives you an instant compact git status-like info in your prompt, tells you if there are changes to pull, whether you’re committing, how many files you have modified/untracked/stashed, etc. and it’s pretty fast. It’s also very customizable with colors and symbols.
If you use the older git-prompt.sh the same change can be done with:
Index: git-prompt.sh
===================================================================
--- git-prompt.sh (revision 4829)
+++ git-prompt.sh (working copy)
@@ -517,8 +517,9 @@
b="\${__git_ps1_branch_name}"
fi
+ surl="$(git config --get remote.origin.url | sed 's|^.*//||; s/.*@//; s/[^:/]\+[:/]//; s/.git$//')"
local f="$w$i$s$u"
- local gitstring="$c$b${f:+$z$f}$r$p"
+ local gitstring="$c$surl$r|$c$b${f:+$z$f}$r$p"Some change since yesterday broke lr_find.
I ran this code yesterday without any problems:
learn.lr_find(start_lr=1e-3, end_lr=5e-1)
---------------------------------------------------------------------------
UnboundLocalError Traceback (most recent call last)
<ipython-input-6-09bf19560237> in <module>()
----> 1 learn.lr_find(start_lr=1e-3, end_lr=5e-1)
~/Code/fastai/fastai/train.py in lr_find(learn, start_lr, end_lr, num_it, **kwargs)
25 cb = LRFinder(learn, start_lr, end_lr, num_it)
26 a = int(np.ceil(num_it/len(learn.data.train_dl)))
---> 27 learn.fit(a, start_lr, callbacks=[cb], **kwargs)
28
29 def to_fp16(learn:Learner, loss_scale:float=512., flat_master:bool=False)->Learner:
~/Code/fastai/fastai/basic_train.py in fit(self, epochs, lr, wd, callbacks)
135 callbacks = [cb(self) for cb in self.callback_fns] + listify(callbacks)
136 fit(epochs, self.model, self.loss_func, opt=self.opt, data=self.data, metrics=self.metrics,
--> 137 callbacks=self.callbacks+callbacks)
138
139 def create_opt(self, lr:Floats, wd:Floats=0.)->None:
~/Code/fastai/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
87 except Exception as e:
88 exception = e
---> 89 raise e
90 finally: cb_handler.on_train_end(exception)
91
~/Code/fastai/fastai/basic_train.py in fit(epochs, model, loss_func, opt, data, callbacks, metrics)
84 cb_handler=cb_handler, pbar=pbar)
85 else: val_losses=None
---> 86 if cb_handler.on_epoch_end(val_loss): break
87 except Exception as e:
88 exception = e
UnboundLocalError: local variable 'val_loss' referenced before assignment
The commit that changed line 86:
commit 6b17961574d738d4fbc8b8b8f6268eebca5f61f9
Author: Ubuntu <sylvain.gugger@gmail.com>
Date: Thu Oct 18 19:29:02 2018 +0000
Metrics are now callbacksThat’s this damn ubuntu, not me 
Will fix ASAP.
1 Like
git config user.name “Sylvain (Ubuntu) Gugger”
Jeremy beat me to it. Note that we renamed data to basic_data because it was sometimes creating weird bugs when we always call our data objects data afterward.
Hey all, I was wondering if anyone was doing the documentation for DynamicUNet, it took me a while to figure out how to use it with torchvision's pretrained Resnet, and I wouldn’t mind creating a simple notebook for it if no one else has started.
3 Likes
Unfortunately that awesome tool got blocked quickly by github - it fires too fast and no way to configure it to slow down 
So I have been researching an alternative and I think I have a solid solution.
I will post more once I have it clear.
Until then here is the updated list to fix:
URL `/vision.tta.html#vision.tta'
Name `tta'
Parent URL https://docs.fast.ai, line 852, col 23
Real URL https://docs.fast.ai/vision.tta.html#vision.tta
Check time 0.962 seconds
Size 4KB
Result Error: 404 Not Found
URL `'https://arxiv.org/abs/1506.01186''
Name `described'
Parent URL https://docs.fast.ai/callbacks.lr_finder, line 662, col 47
Real URL https://docs.fast.ai/%27https://arxiv.org/abs/1506.01186%27
Check time 0.421 seconds
Size 4KB
Result Error: 404 Not Found
URL `/text.qrnn.html#text.qrnn'
Name `qrnn'
Parent URL https://docs.fast.ai/text.learner, line 719, col 578
Real URL https://docs.fast.ai/text.qrnn.html#text.qrnn
Check time 0.165 seconds
Size 4KB
Result Error: 404 Not Found
URL `https://docs.fast.ai/gen_doc.sgen_notebooks#update_notebooks'
Name `update_notebooks'
Parent URL https://docs.fast.ai/gen_doc, line 852, col 4
Real URL https://docs.fast.ai/gen_doc.sgen_notebooks#update_notebooks
Check time 0.548 seconds
Size 4KB
Result Error: 404 Not Found
URL `gen_doc.nbdoc.ipynb'
Name `nbdoc'
Parent URL https://docs.fast.ai/gen_doc.gen_notebooks, line 938, col 119
Real URL https://docs.fast.ai/gen_doc.nbdoc.ipynb
Check time 0.955 seconds
Size 4KB
Result Error: 404 Not Found
URL `gen_doc.convert2html.ipynb'
Name `documentation of convert2html'
Parent URL https://docs.fast.ai/gen_doc.gen_notebooks, line 961, col 33
Real URL https://docs.fast.ai/gen_doc.convert2html.ipynb
Check time 1.120 seconds
Size 4KB
Result Error: 404 Not Found
We are almost in the clear! docs-dev is clear now.
@ashaw, one large issue with doc anchors is that they aren’t unique:
$ grep on_batch_end docs/callback.html
Example: change the values of the hyperparameters (if we don't do it on_batch_end instead)</p>
<h4><a id=on_batch_end></a><code>on_batch_end</code></h4>
<blockquote><p><code>on_batch_end</code>(<code>kwargs</code>:<code>Any</code>)</p>
<h4><a id=on_batch_end></a><code>on_batch_end</code></h4>
<blockquote><p><code>on_batch_end</code>(<code>loss</code>:<code>Tensor</code>)</p>
you probably need to prepend the function name to the anchor to make it unique.
You can see a full report here: https://pastebin.com/YkvDnXcd - primarily anchor issues.
Ahh good catch. Yeah looks like class methods are colliding with functions. Yeah I may have to prepend the class
I must be reading this code wrong but it has me stumped. I apologize if this is a stupid question. I’m looking at the validate method in basic_train and val_losses is a list that holds each loss from the batch and each loss is appended onto the list and from what I understand of the append method, each subsequent loss would be added to the end of val_losses, but then for the callback hander when the on_batch_end is called, it is passing in val_losses[0]. Doesn’t this then always send the same first loss to on_batch_end??? I must be confused. Again, I apologize if this is dumb on my part.
with torch.no_grad():
val_losses,nums = [],[]
for xb,yb in progress_bar(dl, parent=pbar, leave=(pbar is not None)):
if cb_handler: xb, yb = cb_handler.on_batch_begin(xb, yb, train=False)
val_losses.append(loss_batch(model, xb, yb, loss_func, cb_handler=cb_handler))
if not is_listy(yb): yb = [yb]
nums.append(yb[0].shape[0])
if cb_handler and cb_handler.on_batch_end(val_losses[0]): breakSpoke too fast! This is just because lost_batch returns a tuple with one element. I know this is weird and will probably change soon.
The callback handler is called with that val_loss for the batch, then at the end, we average the val_losses found over the batches.