Custom image classifier in 5 min on Mac - what's the alternative on Win/Linux

willismar · December 13, 2018, 3:23pm

If you see on the script the process is this:

1-Grab the arguments from the command line User and UserID
2-Install Ubuntu packages needed
3-Creates your user inside the new container image
4-Put your user inside the container as a SuperUser
5-Creates a working directory
6-Login on the container as your User
7-Download miniconda
8-Put conda on the path
9-Put conda initializaton on the bash script
10-Install all conda packages (PyTorch, Fastai, Jupyter, etc…)
11-Create initialization of Jupyter
12-Create initialization of the standard shell when you try to run the container

If your building stop at any step your image will not be build entirely. So you will need to fix it. Here is working nice. I don’t know exactly why your user cold not been created on Mac. Without fixing this you wont have the correct image available to use.

davide445 · December 13, 2018, 3:30pm

This is what is happening running the build command in Power Shell, I can’t find the error, sorry for my no knowledge of the topic:

Step 8/17 : RUN useradd -Um ${USER}
—> Running in 82f3a56787d5
Usage: useradd [options] LOGIN
useradd -D
useradd -D [options]

Options:
-b, --base-dir BASE_DIR base directory for the home directory of the
new account
-c, --comment COMMENT GECOS field of the new account
-d, --home-dir HOME_DIR home directory of the new account
-D, --defaults print or change default useradd configuration
-e, --expiredate EXPIRE_DATE expiration date of the new account
-f, --inactive INACTIVE password inactivity period of the new account
-g, --gid GROUP name or ID of the primary group of the new
account
-G, --groups GROUPS list of supplementary groups of the new
account
-h, --help display this help message and exit
-k, --skel SKEL_DIR use this alternative skeleton directory
-K, --key KEY=VALUE override /etc/login.defs defaults
-l, --no-log-init do not add the user to the lastlog and
faillog databases
-m, --create-home create the user’s home directory
-M, --no-create-home do not create the user’s home directory
-N, --no-user-group do not create a group with the same name as
the user
-o, --non-unique allow to create users with duplicate
(non-unique) UID
-p, --password PASSWORD encrypted password of the new account
-r, --system create a system account
-R, --root CHROOT_DIR directory to chroot into
-s, --shell SHELL login shell of the new account
-u, --uid UID user ID of the new account
-U, --user-group create a group with the same name as the user
-Z, --selinux-user SEUSER use a specific SEUSER for the SELinux user mapping
–extrausers Use the extra users database

The command ‘/bin/sh -c useradd -Um ${USER}’ returned a non-zero code: 2

willismar · December 13, 2018, 3:31pm

What version of docker do you have ?

Mine is

docker --version
Docker version 18.09.0, build 4d60db4

davide445 · December 13, 2018, 3:32pm

willismar · December 13, 2018, 3:33pm

its the same engine 18.09…

willismar · December 13, 2018, 3:36pm

Try to change this line accordingly and let me know if the build passes that line:

RUN useradd -Um "${USER}"

if not work try this other

RUN useradd -Um "$USER"

davide445 · December 13, 2018, 3:37pm

I suppose the problem is in the

ENV USER=${user}

Statement, since running it in Power Shell got an error. Can’t find how to set the user and userid in Windows Power Shell.

willismar · December 13, 2018, 3:38pm

this line works like this …
It creates a environment variable inside your container by capturing the “user” variable from your --build-arg user=XXXXX

docker build --build-arg user=$USER --build-arg uid=$UID -t fastai:latest .

davide445 · December 13, 2018, 3:40pm

With both your options the result is

Step 8/17 : RUN useradd -Um “$USER”
—> Running in 7be90e399d49
useradd: invalid user name ‘’
The command ‘/bin/sh -c useradd -Um “$USER”’ returned a non-zero code: 3

willismar · December 13, 2018, 3:41pm

OK are you in MacOs or Windows ?

I am not familiar to the term PowerShell on MacOs. The MacOS has linux as the basis system (Darwin)

davide445 · December 13, 2018, 3:45pm

I’m on Windows 10, Power Shell is a command line scripting shell in Windows.
I use it only since I think is more powerful than CMD and “maybe” better supported, didn’t know anything about it.

I have in this laptop also a virtual environment with Ubuntu Budlge installed, problem is can’t connect on Internet using my laptop connection, didn’t know why.

willismar · December 13, 2018, 3:46pm

Do you have the ubuntu bash on windows 10 ?

willismar · December 13, 2018, 3:51pm

OK windows is a entire different beast
because what is created in linux not influence the files on windows , in case of linux and macos without creating your user inside the container you coun’t share your files without permission.

I will change the docker script so you don’t need to use a user for it.

willismar · December 13, 2018, 4:00pm

Windows Version (CPU-Only) [UPDATED]

Dockerfile

FROM ubuntu:16.04

ARG PYTHON_VERSION=3.7
RUN apt-get update && apt-get install -y --no-install-recommends \
         sudo \
         build-essential \
         cmake \
         git \
         curl \
         vim \
         ca-certificates \
         libjpeg-dev \
         libpng-dev \
         net-tools \
         iproute2 && \
     rm -rf /var/lib/apt/lists/*

RUN curl -o ${HOME}/miniconda.sh -O  https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh  && \
     chmod +x ${HOME}/miniconda.sh && \
     ${HOME}/miniconda.sh -b -p ${HOME}/conda && \
     rm ${HOME}/miniconda.sh

ENV PATH ${HOME}/conda/bin:${PATH}

RUN echo "export PATH=${HOME}/conda/bin:${PATH}" | tee -a ${HOME}/.bashrc

RUN  ${HOME}/conda/bin/conda install -y python=${PYTHON_VERSION} jupyter notebook && \
     ${HOME}/conda/bin/conda install -y numpy mkl mkl-include setuptools cmake cffi typing pyyaml scipy mkl mkl-include cython typing pip && \
     ${HOME}/conda/bin/conda install -y -c pytorch pytorch-nightly-cpu torchvision-cpu && \
     ${HOME}/conda/bin/conda install -y -c mingfeima mkldnn && \
     ${HOME}/conda/bin/conda install -y -c fastai fastai

RUN  mkdir /projects -p && \
     echo "export JUPYTER_IP=\"$(ip route get 8.8.8.8 | awk '{print $NF; exit}' )\""  | tee -a ${HOME}/.bashrc && \
     echo "export JUPYTER_PORT=8888"  | tee -a ${HOME}/.bashrc && \
     echo 'alias jbook="jupyter notebook --port=${JUPYTER_PORT} --ip=${JUPYTER_IP}  --NotebookApp.notebook_dir=/projects --no-browser --allow-root"'  | tee -a ${HOME}/.bashrc && \
     echo "source activate" | tee -a ${HOME}/.bashrc

CMD "/bin/bash"

Build command

docker build -t fastai:latest .

Running the container
1-In case you don’t have a sharing folder between windows and container:

docker run -it --rm -p 8888:8888  fastai:latest bash

2-In case you have a sharing folder between windows and container:
Note: assuming c:\projects

docker run -it --rm -p 8888:8888 -v c:/projects:/projects fastai:latest bash

Take a look how to create a folder to share with docker here:

Running Jupyter
Type on the shell:

jbook

davide445 · December 13, 2018, 4:12pm

I think now is working with this version, I’m seeing it compiling the various packages

willismar · December 13, 2018, 4:12pm

It may be interested to you to enable this Experimental features on docker

Doing this you can reduce the size of the image container. But you don’t need to bother with this.

davide445 · December 13, 2018, 4:22pm

Successfully finished!

Successfully tagged fastai:latest
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and reset permissions for sensitive files and directories.

I suppose I need to change Docker from Linux like to Windows lilke

willismar · December 13, 2018, 4:23pm

@davide445 congratulations

The whole thing about user permission on linux and mac is this

I just made a test on Linux here to show you:

I have a folder projects in my user Desktop

total 8,0K
-rw-r--r-- 1 root      root       573 Dez 13 14:21 NewJupyterFileCreatedOnContainer.ipynb

after I start and run the docker container and run the jupyter sharing a folder with my linux (from linux to linux) anything I create inside that shared folder from the container without my own user, will be created as ROOT user. So outside the container in the HOst machine I loose access to the file as my default user because linux doesnot allow users touch each other files. Different from windows.

Normally I would have this:

total 8,0K
-rw-r--r-- 1 willismar  willismar     573 Dez 13 14:21 NewJupyterFileCreatedOnContainer.ipynb

That’s why the process may work in windows because windows cannot control the linux permissions.

davide445 · December 13, 2018, 4:55pm

Ok thanks again for your wonderful work!
Having the image up and running, just a (first) stupid question: how I test if fast.ai v1 and Pytorch are working correctly?

willismar · December 13, 2018, 4:57pm

you can just create a Jupyter notebook and put this

import torch
print(torch.__version__)

import fastai
print(fastai.__version__)

if the packages can be loaded they can be used.