Python help

Let’s use this thread to discuss learning python!

6 Likes

Python basics

a nice reminder of the basics of python for starters

What does *args and **kwargs mean?

How to explore a python object?

How to make the most out of pdb to inspect code?

Jeremy’s book recommendation on learning pandas

Another useful book online about python and problem solving

1 Like

What are *args and **kwargs?

*args: a tuple of arguments whose number is not fixed
**kwargs: a dictionary of an unknown amount arguments who have keys and values

The two examples (I copied from here) will illuminate how they are used

def adder(*num):
    sum = 0
    
    for n in num:
        sum = sum + n

    print("Sum:",sum)

adder(3,5)
adder(4,5,6,7)
adder(1,2,3,5,6)

The output is

Sum: 8
Sum: 22
Sum: 17

def intro(**data):
    print("\nData type of argument:",type(data))

    for key, value in data.items():
        print("{} is {}".format(key,value))

intro(Firstname="Sita", Lastname="Sharma", Age=22, Phone=1234567890)
intro(Firstname="John", Lastname="Wood", Email="johnwood@nomail.com", Country="Wakanda", Age=25, Phone=9876543210)

The output is

Data type of argument: <class 'dict'>
Firstname is Sita
Lastname is Sharma
Age is 22
Phone is 1234567890

Data type of argument: <class 'dict'>
Firstname is John
Lastname is Wood
Email is johnwood@nomail.com
Country is Wakanda
Age is 25
Phone is 9876543210

2 Likes

How to explore a python object?

1 Like

I’ve moved your python posts to a separate threads @Daniel, since they were previously in an Excel topic. Hope that’s OK!

1 Like

Folks interested in learning might want to start here: Free intro to coding in python kaggle course

A nice resource to learn Python and an even better community to put that learning into action :slight_smile:

2 Likes

Sure, thanks Jeremy, this is much nicer! This way more people can know, contribute and benefit from it.

Great initiative @daniel. Do you know any good resources summarising the dunder-builtins? I presume some of those are data and some functions?

Challenge: A code snippet that iterates the dir() list to summarise:

  • data displays its type, and maybe its display string
  • functions display the function name, and bonus also the function signature

Super Challenge: Output consists of links that can be clicked to auto-fill the next cell and execute it, so you can drill down into the data structure with mouse clicks.

1 Like

Not exactly sure whether I understand what you mean here, but it forced me to dig into my previous notes on pdb from my last life :wink: in fastai which I totally forgot almost. (please tolerate a few Chinese characters there, I was saving them in case I forgot which I did. ) I will give a try to them later to see whether they can in jupyter notebook or not.



Actually I don’t think you need to download pdbpp for using the aliases above in Jupyter notebook. Now I remembered pdbpp is very nice (coloring and other conveniences) to work on pure pyfile but not Jupyter notebook. If you can get it to work with Jupyter notebook, I’d love to learn to use it there too.

Yes, they still work I just tried them in jupyter notebook

2 Likes

What is the hidden secret function of *args?

My question is in the code block below

# merge uses *args
merge(*[{'lr': 0.01}, {'lr2': 0.001}], {'mom': 0.9}, {'lr2': 0.1})

# * turns all the inputs above into a tuple below before running the source code of merge
({'lr': 0.01}, {'lr2': 0.001}, {'mom': 0.9}, {'lr2': 0.1})

# What exactly does * do to all the input args of the function merge

The basic info of merge:

Type: function
String Form: <function merge at 0x14247a0d0>
File: /Users/Natsume/mambaforge/lib/python3.9/site-packages/fastcore/basics.py
Definition: merge(*ds)
Docstring: Merge all dictionaries in ds

Source code of merge is the following:

def merge(*ds):
	 "Merge all dictionaries in `ds`"
	 return {k:v for d in ds if d is not None for k,v in d.items()}

How to do multi-line plotting with matplotlib

This is a basic template to play with.

def plot_multlines(line1, # Tensor|np.array, 
                   line2, # Tensor|np.array
                  ):
    if type(line1.data) == torch.Tensor: 
        line1 = line1.data.numpy()[0]
        
    assert type(line1) == np.ndarray, f"Make use line1 to be a tensor or numpy array"
    
    if type(line2.data) == torch.Tensor: 
        line2 = line2.data.numpy()[0]

    assert type(line2) == np.ndarray, f"Make use line2 to be a tensor or numpy array"
    
    fig1 = plt.figure(1)

    ax = fig1.add_subplot()

    ax.plot(line1, color="orange", linestyle="solid", label="pure");
    ax.plot(line2, color="blue", linestyle="dashed", marker="o", drawstyle="steps-post", alpha=0.5, label="sgd_step");
    ax.legend()
    ticks = ax.set_xticks([0, 10, 20, 30])
    labels = ax.set_xticklabels(["0", "10", "20", "30"], rotation=30, fontsize=8)
    ax.set_xlabel("iterations")
    ax.set_ylabel("weights")
    ax.set_title("Stepping effects on weights")
    plt.rc("figure", figsize=(30, 30))

Hi @Moody , I was reading the source of fastcore.delegates and find the behavior of this line is quite interesting. So, I experiment it a little and could you help to verify my understanding of it?

This is the source code

else: to_f,from_f = to.__init__ if isinstance(to,type) else to,f

Below is the experiment I used to help understand the source. My suspicion is that the two blocks of codes below are equivalent. Am I right?

to = 1
f = 34
if True: to_f,from_f = 3 if to == 1 else to,f 
print(to_f, from_f, f)

A longer but equivalent one is

to = 1
f = 34
if True: 
    if to == 1: 
        to_f = 3
        from_f = f
    else: 
        to_f = to
        from_f = f
print(to_f, from_f, f)

What is the best or easiest way to find out how many functions are there in a library such as fastcore or fastai with python?

I have tried to import fastcore as core and from fastcore import * and used __dict__ and dir() and locals(), but got nothing useful.

Could anyone help? Thanks

There isn’t one easy way in general. But fastcore has fastcore.all so you can import that, then look at fastcore.all.__all__.

2 Likes

Yes, you are right. The two blocks are doing the same things.

I turned your code blocks into functions (cells 1 & 2). So, it is easier to pass different inputs and check their results (cells 3 & 4).

You can also use assert to evaluate their results if they are identical using ==. So, you can test different input scenarios easily. see Python Testing for more information. If everything is doing fine, no output for that cell (cells 5-7).

I intended to make an input error for cell 8. When the results are different, it returned AssertionError. For your information, there are many sub-type of the AssertionError and you can treat them differently.

1 Like

I have tried it in ipython, but couldn’t find fastcore.all.__all__.

But I can do it in another way with fastcore.all:

import fastcore.all as fa
all = dir(fa)
print(len(all)) # 368

all_func = [getattr(fa, i) for i in all]
count = 0
for i in all_func:
    if callable(i):
        count = count + 1
        
count # 327

Thank you Sarada, these are all very helpful!

1 Like

Ah yes sorry I forgot fastcore.all doesn’t have a __all__. All the submodules do however. Good job finding another way!

3 Likes

Thanks Jeremy! Here is the function I use to find out how many callables in a module and how many items are included inside module.__all__ if available. But compared with __all__, checking out all the callables isn’t very useful after all.

def countFunc(mo, # module, e.g., `import fastcore.all as fa`, use `fa` here
              dun:bool=False, # print items in __all__
              cal:bool=False # print all callables
             ): 
    'count how many callables are in the module and how many items included in __all__'
    full = dir(mo)
    all_func = [getattr(mo, i) for i in full]
    count = 0
    for i in all_func:
        if callable(i):
            count = count + 1
        else:
            all_func.remove(i)
    num_all = 0 if hasattr(mo, "__all__") == False else len(mo.__all__)
    print(f"there are {count} callables in {mo.__name__} and {num_all} items in its __all__")
    if hasattr(mo, "__all__") == True and dun: pprint(mo.__all__)
    if cal: pprint(all_func)

Now, I have upgraded this function to be able to tell you everything you may want to know about a module.

def whatinside(mo, # module, e.g., `import fastcore.all as fa`, use `fa` here
               dun:bool=False, # print all items in __all__
               func:bool=False, # print all user defined functions
               clas:bool=False, # print all class objects
               bltin:bool=False, # print all builtin funcs or methods
               lib:bool=False, # print all the modules of the library it belongs to
               cal:bool=False # print all callables
             ): 
    'Check what inside a module: __all__, functions, classes, builtins, and callables'
    dun_all = len(mo.__all__) if hasattr(mo, "__all__") else 0
    funcs = getmembers(mo, isfunction)
    classes = getmembers(mo, isclass)
    builtins = getmembers(mo, isbuiltin)
    callables = getmembers(mo, callable)
    pkgpath = os.path.dirname(mo.__file__)
    print(f"{mo.__name__} has: \n{dun_all} items in its __all__, and \n{len(funcs)} user defined functions, \n{len(classes)} classes or class objects, \n{len(builtins)} builtin funcs and methods, and\n{len(callables)} callables.\n")  
    if hasattr(mo, "__all__") and dun: pprint(mo.__all__)
    if func: 
        print(f'The user defined functions are:')
        pprint([i[0] for i in funcs])
    if clas: 
        print(f'The class objects are:')
        pprint([i[0] for i in classes])
    if bltin: 
        print(f'The builtin functions or methods are:')
        pprint([i[0] for i in builtins])
    if cal: 
        print(f'The callables are: ')
        pprint([i[0] for i in callables])
    if lib: 
        modules = [name for _, name, _ in pkgutil.iter_modules([pkgpath])]
        print(f'The library has {len(modules)} modules')
        pprint(modules)


2 Likes