Student project: LEGO and Computer Vision

Part 1 (2020)
1

Dear all,

This is an idea for an open-source project:

  • count the number of lego bricks in a lego building
  • and divide per color:

image
image1500×1483 409 KB

we could start from this dataset:

App
INPUT (video of the lego building)
OUTPUT: Count of the Lego bricks and division per color

Supported by Computational Design Institute (501c3 nonprofit in California)

Duration: 12months
Phases:

[Deep Learning Part 1]

  1. Dataset analysis
  2. Recognize LegoBrick
  3. Recognize Different Colour

[Deep Learning Part 2]

  1. Recognize Separate Bricks (The Construction will be make by the same type of Bricks )
    image
  2. Using 6 different bricks, [harder]
    image
  1. Deploy on a device (iPad Pro)
1 Like
2

Making some progress on the research side, I have found an interesting research paper that would be nice to translate to pytorch and fastai as a project. It uses fully connected deep network and MLP.
https://arxiv.org/pdf/2003.08934.pdf Neural Radiance Fields of View

here the code https://github.com/bmild/nerf

and here the video
https://www.youtube.com/watch?v=JuH79E8rdKc&feature=youtu.be

image

3

This is cool. I’m interested. What is your final aim though? to build 3D models of the construction like the nerf paper or just counting bricks during construction?

1 Like
4

The aim is to achieve both possibly.
I would be personally happy if we arrive to an approximation of the count. But it will be a great to achieve a full 3d model. I am a newbie, but pretty passionate about the subject and with 1 year and half head to develop this project. It will be crucial also to try the integration with Pytorch3D @nchukaobah.

5

I’m unfamiliar with Pytorch3D. I’ll check it up later this week. I’ll also check out the nerf paper but that might have to wait for next week.

Have you done any work on identifying the blocks? I can work on that this week, maybe put up a github page to chart my progress.I’m thinking, with the dataset, a typical object detection model can work. Or even simple color detection algorithms if color is a strong feature for a block. What do you think?

6

Thanks @nchukaobah we can use this repository as a starting point, https://github.com/CDInstitute/LEGO so we can tailor with the initiative of the Computational Design Insitute as a non-profit. Looking forward to work on this with all of you.

7

Hey @Albertotono, I’m so busy right now so I won’t be able to keep up with the project. Have you been able to make progress?

8

@nchukaobah, @Albertotono, you might check out the LEGO sorter project that @pryb posted. It’s a very cool project and you might find some interesting inspirations there.

1 Like