Controlling a Drone

I’ve started a project to control a drone using various machine vision techniques by sending captured frames to be processed and decide how to control the drone. I have managed to hook up dlib with the drone and it is now following me around by my face.

The next step is integrating a depth perception model that I’ve been working on however I’m having a problem with the images coming in from the stream.


The UDP reads a stream and creates a np array but I can’t seem to get a fastai Image created properly from it.

Here is a sample of the array that gets created


n = np.load('test.npy')
# Convert from BGR
n = n[:,:,::-1]
p = PIL.Image.fromarray(n)'p1.jpg')
t = pil2tensor(p, dtype=np.uint8)
im = Image(t)'p2.jpg')

Any suggestions for possible things to do or if you interested in contributing, here is the Repository

Drone Model


Solved the problem by using:

img = arr[:,:,::-1].transpose((2,0,1)).copy()
img = torch.from_numpy(img).float().div(255.0)

By curious, so you are trying to guess the depth from RGB image ? I have image from with depth information (Intel Realsense camera), do you know some model that integrate the depth ? Thank you

Yes please look at my post on depth perception:

1 Like

Great ! I will read your post. Thanks :smiley:

Would probably be a good task to also replace the dlib model with one created with fastai. There are many datasets for this problem

Depth Perception Demo

Results aren’t too bad. Definitely could create some logic to use the results to move the drone to different areas.

Potential Next Steps:

  1. Take the processed results and ask the drone to fly towards the piece with the highest 5 x 5 average CNN pool. i.e. the direction which is least explored whilst making sure that the average of the resulting tensor is greater than a certain amount (Flying into walls is bad).

  2. Map out a room (or do as good of a job as possible), this could be possible if the drone knows how much it is moving around but I’m not sure the results are good enough.