šŸ‘ Training a Network to Detect Hands

Deep Learning
21

You can train your network by tiny-cnn/dlib directly if you like, I have trained a car vs non-car classifier by tiny-cnn before, and use dlib to participate dog vs cat competition, both of them are sufficient to describe your network architectures even they are not as mature as another deep learning frameworks like PyTorch, TensorFlow nor Keras. Unfortunately, compare with Python, deep learning libraries of c++ are far from mature(lack of features).

Another solution is import the model by opencv3.3 like I mentioned before, since you are target on wins and mac ox, binary size and performance should not be an issue.

22

I thought about it, but I really want to benefit from transfer learning using a few layers of VGG or Mobilenet in my network, which worked really well so far! I do not have a huge amount of training data: experimenting with pretrained networks and finetuning the network in keras/pytorch seems more efficient to me.

If only I could fast-forward 6 months and ONNX was already supported by every framework :grin:

I did not know about this, thank you! :heart_eyes: That actually looks promising since Iā€™m already including OpenCV3.X in my app! OpenCV-DNN is what you are using in your app, right? How is performance?

Thank you for the offer! :blush: So far there isnā€™t much of a ā€œprojectā€ yet, Iā€™m mostly doing the research to see if I can leverage the power of DNNā€™s in games.

And then you mean Lua torch, not pytorch, correct? Perhaps I can export from pytorch to a caffe model and import that in OpenCV-DNNā€¦

23

Correct, I really like the api and flexibility of PyTorch(http://pytorch.org/), but opencv3.3 do not support PyTorch(http://pytorch.org/) but pytorch(https://github.com/hughperkins/pytorch), you can study more details in #9501.

I do not like caffe nor tensorflow but I would prefer them if I want to deploy my model, because I canā€™t find a better choice.

Right, opencv dnn is what I use, you can check the source codes(ssd_detector.cpp, less than 90 lines of codes), if there are anything you do not understand, I am willing to to help.

I think performance is quite good, SSDMobileNet can run 70ms per frame on my laptop(Y410P) without using gpu(I link with the opencv compile without cuda support and disable opencl from the beginning, but I build with eigen and tbb support), however, dnn module do eat a lot of cpu power(80~90% cpu usage).

Hope you success, if you want to find someone to cooperate and you think my skills may help you, please do not hesitate to send me a message, I am eager to earn more experiences of computer vision project.

1 Like
24

I donā€™t know what the desired target platform is, but to give another datapoint: MobileNet+SSD runs at 30 FPS on iPhone 7 using the GPU (latency is about 50ms in that case). If you want good performance on mobile, OpenCV (or even TensorFlow) is way slower than a native solution.

25

What framework is that? By ā€œnativeā€ do you mean with CoreML?

My target platform is probably an Intel NUC, or another portable PC that runs the tracking server and sends found objects to a different PC running a Unity game (over OSC).

26

That is using Metal. Core ML tends to be slower than Metal in my experience. Doing it in OpenGL would also work, if you had a fast implementation for the convolution layers.

For the Intel NUC you might want to look into Intelā€™s clDNN library https://github.com/01org/cldnn ā€“ although it might not work on the GPU inside the NUC. But at least OpenGL should work on it.

1 Like
27

I think your comparison is unfair.

  1. You are comparing iphone7 with my laptop
  2. You are comparing gpu vs pure cpu(opencv dnn support gpu too)

By the way, MobileNetSSD is much more complicated than the network architecture @noio want to build.

Performance of opencv dnn(3.3)

SqueezeNet only took 4.96ms per frame on cpu, I guess this is more than enough for @noio app.

opencv dnn already implement the network with clDNN

1 Like
28

I didnā€™t interpret it as a direct comparison, just ā€œanother datapointā€. :slight_smile:

I also think (and hope) that I can find a CPU based solution that will suit my needs. As you said, what I need is much less than a full SSD. As you can see in one of my first post, I only used 8 Conv layers with good results.

Additionally, ideally Iā€™d like the executable to be somewhat portable, so including opencv-dnn or tiny-dnn is ideal in that case (instead of relying on a platform specific clDNN / Metal ) Obviously, the more you make use of a platformā€™s characteristics, the faster a solution will be. Exploiting Intel graphics is faster than running on a CPU. And exploiting a CPUā€™s BLAS/AVX will be faster than a naive solution.

29

In this case it make sense, @machinethink mentioned that too, my bad, misunderstand his intentions, sorry for the confusion.

You can compile your apps as different binary if needed. Since you are using unity, this will be a 3D game?

1 Like
30

Yeah, it wasnā€™t my intention to compare your results to mine, since weā€™re using totally different computers. Sorry if that wasnā€™t clear :slight_smile:

I only wanted to point out that something like OpenCV on iOS wonā€™t have the same performance as Metal. On the desktop itā€™s a totally different story.

Unfortunately, for mobile devices there isnā€™t a cross-platform solution right now, except for writing your own kernels in OpenGL. (Unless there already is an OpenGL-based DNN library?)

31

Maybe you already did this, I think you can augment your data by

1 : Separate humans from background
2 : Put humans into different background

Step 1 could be done without too much headache if you take the photo/video with white background, else you may need to use some algo to separate human from background(tiramitsu, use your hand separation net to separate human, use graph cut to manually cut out human etc).

32

So, first update in a long time!

Currently, the training procedure (including data/labeling) I consider solved. I have trained simple network built on top of tiny-YOLO that gets the accuracy I need. I can come back to this later. Iā€™ve exported the network from PyTorch to Caffeā€™s format and (after some pains) managed to import it into tiny-dnn.

The problem right now is achieving real-time performance in an easy-to-deploy executable. I would like for it to run on different hardware (including my macbook) with minimal installation.

@tham I am impressed by the 70ms you achieve CPU only, because when I naively load my network (8 conv blocks) into tiny-dnn with NNPack, it takes up to 6000ms per frame. Do you use OpenCVā€™s dnn-modern (which is a wrapper around tiny-dnn ? What else did you do to optimise?

EDIT: Hahahah. :joy: I simply forgot to set the right backend in tiny-dnn using layer>set_backend_type(backend_t::nnpack);. Went from 6000ms to 350ms, but would like to optimise further.

33

I do not use dnn-modern, only opencv dnn module.

Nothing(I compile opencv with intelTBB). I import the model and forward it as the example shown(you can study my source codes). I believe opencv dnn module do aggressive optimization on cpu part, after all it is a project back by intel :slight_smile:.

If you want to save some pain, you can use torch or caffe to train your 8 layers network, I described squeezeNet1.1 by torch and import it by opencv dnn without any pain(almost).

I do not recommend tensorflow if you plan to export train model to opencv dnn, because

  1. opencv dnn do not have good support on tensorflow yet, there are many surprise
  2. tensorflow is an over complicated library, better stay away from it unless you cannot find better candidate or your boss ask you to do that

Although my experiences of lua is close to zero, I find out torch still much more easier to learn and use compare with tensorflow, it is a well designed library + you can import trained model by opencv dnn with zero pain(almost). Right now opencv do not support nngraph by torch, I guess they are busy on fixing the issues related to tensorflow(as Jeremy said, tensorflow is too complicated)

Following are the things you need to take care when saving torch model

--clear the state first, else model size will be ultra big and trained batchnorm
--may not be used correctly
net:clearState()
--convert model from gpu to cpu, else save model cannot be loaded by opencv dnn
net = net:float()
torch.save('squeeze_homo', net)

After that, just load it by opencv dnn.

dnn::Net net = dnn::readNetFromTorch(model);

Very simple, isnā€™t it?If you need yolo, maybe this post(#9705, port yolo v2 to opencv) can help you(unless I am quite interesting).

ps : You can change your network architectures from vgg like architectures to squeezeNet or mobileNet like.

34

Thanks for the info!

Iā€™m having some trouble finding any documentation on OpenCV-DNN, for example: does it support Depth-Separable Convolutions (that mobilenet needs?)

EDIT: Found docs! Itā€™s all still brand new :slight_smile: https://github.com/opencv/opencv/wiki/Deep-Learning-in-OpenCV

EDIT2: Changing the version of OpenCV thatā€™s inside openFrameworks (to 3.3.0) is also terrible >.<. But the default included OpenCV 3.1 doesnā€™t include the DNN module yet.

(P.S. Iā€™m pretty happy with my PyTorch workflow, so I would prefer to stick with that. But OpenCV-DNN can import Caffe so that seems OK!)

1 Like
35

Hi. I am trying to do something similar and i like the simplicity in your approch.

Can u share your code with me?

Wolfsonavi@gmail.com

36

@aviwolfson25

1 Like