1 Like
I’ve moved your python posts to a separate threads @Daniel, since they were previously in an Excel topic. Hope that’s OK!
1 Like
Folks interested in learning might want to start here: Free intro to coding in python kaggle course
A nice resource to learn Python and an even better community to put that learning into action 
2 Likes
Sure, thanks Jeremy, this is much nicer! This way more people can know, contribute and benefit from it.
Great initiative @daniel. Do you know any good resources summarising the dunder-builtins? I presume some of those are data and some functions?
Challenge: A code snippet that iterates the dir() list to summarise:
- data displays its type, and maybe its display string
- functions display the function name, and bonus also the function signature
Super Challenge: Output consists of links that can be clicked to auto-fill the next cell and execute it, so you can drill down into the data structure with mouse clicks.
1 Like
Not exactly sure whether I understand what you mean here, but it forced me to dig into my previous notes on pdb from my last life 
in fastai which I totally forgot almost. (please tolerate a few Chinese characters there, I was saving them in case I forgot which I did. ) I will give a try to them later to see whether they can in jupyter notebook or not.
Actually I don’t think you need to download pdbpp for using the aliases above in Jupyter notebook. Now I remembered pdbpp is very nice (coloring and other conveniences) to work on pure pyfile but not Jupyter notebook. If you can get it to work with Jupyter notebook, I’d love to learn to use it there too.
Yes, they still work I just tried them in jupyter notebook
2 Likes
What is the hidden secret function of *args?
My question is in the code block below
# merge uses *args
merge(*[{'lr': 0.01}, {'lr2': 0.001}], {'mom': 0.9}, {'lr2': 0.1})
# * turns all the inputs above into a tuple below before running the source code of merge
({'lr': 0.01}, {'lr2': 0.001}, {'mom': 0.9}, {'lr2': 0.1})
# What exactly does * do to all the input args of the function merge
The basic info of merge:
Type: function
String Form: <function merge at 0x14247a0d0>
File: /Users/Natsume/mambaforge/lib/python3.9/site-packages/fastcore/basics.py
Definition: merge(*ds)
Docstring: Merge all dictionaries inds
Source code of merge is the following:
def merge(*ds):
"Merge all dictionaries in `ds`"
return {k:v for d in ds if d is not None for k,v in d.items()}
How to do multi-line plotting with matplotlib
This is a basic template to play with.
def plot_multlines(line1, # Tensor|np.array,
line2, # Tensor|np.array
):
if type(line1.data) == torch.Tensor:
line1 = line1.data.numpy()[0]
assert type(line1) == np.ndarray, f"Make use line1 to be a tensor or numpy array"
if type(line2.data) == torch.Tensor:
line2 = line2.data.numpy()[0]
assert type(line2) == np.ndarray, f"Make use line2 to be a tensor or numpy array"
fig1 = plt.figure(1)
ax = fig1.add_subplot()
ax.plot(line1, color="orange", linestyle="solid", label="pure");
ax.plot(line2, color="blue", linestyle="dashed", marker="o", drawstyle="steps-post", alpha=0.5, label="sgd_step");
ax.legend()
ticks = ax.set_xticks([0, 10, 20, 30])
labels = ax.set_xticklabels(["0", "10", "20", "30"], rotation=30, fontsize=8)
ax.set_xlabel("iterations")
ax.set_ylabel("weights")
ax.set_title("Stepping effects on weights")
plt.rc("figure", figsize=(30, 30))
Hi @Moody , I was reading the source of fastcore.delegates and find the behavior of this line is quite interesting. So, I experiment it a little and could you help to verify my understanding of it?
This is the source code
else: to_f,from_f = to.__init__ if isinstance(to,type) else to,f
Below is the experiment I used to help understand the source. My suspicion is that the two blocks of codes below are equivalent. Am I right?
to = 1
f = 34
if True: to_f,from_f = 3 if to == 1 else to,f
print(to_f, from_f, f)
A longer but equivalent one is
to = 1
f = 34
if True:
if to == 1:
to_f = 3
from_f = f
else:
to_f = to
from_f = f
print(to_f, from_f, f)
What is the best or easiest way to find out how many functions are there in a library such as fastcore or fastai with python?
I have tried to import fastcore as core and from fastcore import * and used __dict__ and dir() and locals(), but got nothing useful.
Could anyone help? Thanks
There isn’t one easy way in general. But fastcore has fastcore.all so you can import that, then look at fastcore.all.__all__.
2 Likes
Yes, you are right. The two blocks are doing the same things.
I turned your code blocks into functions (cells 1 & 2). So, it is easier to pass different inputs and check their results (cells 3 & 4).
You can also use assert to evaluate their results if they are identical using ==. So, you can test different input scenarios easily. see Python Testing for more information. If everything is doing fine, no output for that cell (cells 5-7).
I intended to make an input error for cell 8. When the results are different, it returned AssertionError. For your information, there are many sub-type of the AssertionError and you can treat them differently.
1 Like
I have tried it in ipython, but couldn’t find fastcore.all.__all__.
But I can do it in another way with fastcore.all:
import fastcore.all as fa
all = dir(fa)
print(len(all)) # 368
all_func = [getattr(fa, i) for i in all]
count = 0
for i in all_func:
if callable(i):
count = count + 1
count # 327
Thank you Sarada, these are all very helpful!
1 Like
Ah yes sorry I forgot fastcore.all doesn’t have a __all__. All the submodules do however. Good job finding another way!
3 Likes
Thanks Jeremy! Here is the function I use to find out how many callables in a module and how many items are included inside module.__all__ if available. But compared with __all__, checking out all the callables isn’t very useful after all.
def countFunc(mo, # module, e.g., `import fastcore.all as fa`, use `fa` here
dun:bool=False, # print items in __all__
cal:bool=False # print all callables
):
'count how many callables are in the module and how many items included in __all__'
full = dir(mo)
all_func = [getattr(mo, i) for i in full]
count = 0
for i in all_func:
if callable(i):
count = count + 1
else:
all_func.remove(i)
num_all = 0 if hasattr(mo, "__all__") == False else len(mo.__all__)
print(f"there are {count} callables in {mo.__name__} and {num_all} items in its __all__")
if hasattr(mo, "__all__") == True and dun: pprint(mo.__all__)
if cal: pprint(all_func)
Now, I have upgraded this function to be able to tell you everything you may want to know about a module.
def whatinside(mo, # module, e.g., `import fastcore.all as fa`, use `fa` here
dun:bool=False, # print all items in __all__
func:bool=False, # print all user defined functions
clas:bool=False, # print all class objects
bltin:bool=False, # print all builtin funcs or methods
lib:bool=False, # print all the modules of the library it belongs to
cal:bool=False # print all callables
):
'Check what inside a module: __all__, functions, classes, builtins, and callables'
dun_all = len(mo.__all__) if hasattr(mo, "__all__") else 0
funcs = getmembers(mo, isfunction)
classes = getmembers(mo, isclass)
builtins = getmembers(mo, isbuiltin)
callables = getmembers(mo, callable)
pkgpath = os.path.dirname(mo.__file__)
print(f"{mo.__name__} has: \n{dun_all} items in its __all__, and \n{len(funcs)} user defined functions, \n{len(classes)} classes or class objects, \n{len(builtins)} builtin funcs and methods, and\n{len(callables)} callables.\n")
if hasattr(mo, "__all__") and dun: pprint(mo.__all__)
if func:
print(f'The user defined functions are:')
pprint([i[0] for i in funcs])
if clas:
print(f'The class objects are:')
pprint([i[0] for i in classes])
if bltin:
print(f'The builtin functions or methods are:')
pprint([i[0] for i in builtins])
if cal:
print(f'The callables are: ')
pprint([i[0] for i in callables])
if lib:
modules = [name for _, name, _ in pkgutil.iter_modules([pkgpath])]
print(f'The library has {len(modules)} modules')
pprint(modules)
2 Likes
Hi @jeremy
I am exploring the source on _mk_param and use_kwargs_dic of fastcore, and like the idea of forcing Parameter to change kind from POSITIONAL_OR_KEYWORD to KEYWORD_ONLY. I think this may help to solve a tiny problem of fastcore.meta.delegates I noticed when exploring it.
The problem is presented in the first image below, and the solution learnt from _mk_param and use_kwargs_dict is in the second image.
What do you think of it? Thanks!
1 Like
Thanks Jeremy, I will give it try to submit a PR with it.