@aidank A bolt on attachment to a tractor would be easier but Im excited about having a low powered device that I can leave in a field for hours on end guided using only GPS and a laptop.
@jerevon Thanks, how is your project going? Do you have a machine in the fields yet?
Deleted video
it’s going well, we try to shrink deep neural model to speed up prediction. we currently do not have machine in the fields as developing software is our priority.
Sounds interesting it’s a great practical problem to be tackling now. If you ever need a rover to mount you software and cameras let me know. 
that’s really cool. Thanks.
Ok things are starting to move along here and the prototype test rig is now just about finished. I am now about to start the programming side of things. Its winter here so I am thinking I can set up a test course inside one of the farm sheds as the fields have sheep on them now and no weeds. I have a space with a nice concrete floor I can scatter some bright objects on the floor like some yellow rubber ducks or painted blocks of wood and build a classifier to detect them against the concrete floor. I’m hoping that the bright colour contrast will mean the ducks will be accurately and easily detected.
Im new to coding so my first challenge is sending data to the Arduino from Python in the form of an array to switch off and on the spray heads. If anyone wants to help please do 
I will post what I have on a Github account.
Gav
Sprayer boom, spray heads and hoses fitted, starting to look like it might be useful.
Each sprayer has its own 12v relay to control the flow of water. I can use python to send a command over the serial port to an Arduino which can turn them on and off using a standard 8 piece relay board you can buy off Ebay, etc.
Now that I have the Arduino serial bridge between python and the real world (spray heads). I need to find a way to run the CNN on a laptop, give me classifications on each section of the field image and then store it in an array. I hope to then pass this array/string of classifications which would look something like a string of 1’s and 0’s to the Arduino. The Arduino would then use the encoder on the wheel to work out where it is in relation to the image it just took and turn the relays off and on when needed. Once its worked its way over the first image it will send a signal back to python to take another image and the process begins again.
The nice thing about the electric motor is the rover can stop and take a break between images to think about its next move and crunch some numbers. Long term this is a slow way to go about things but for prototyping its definitely makes it more achievable in terms of getting a nice sharp image from the camera and classifying weeds on a laptop.
I saw this today and thought of your project https://www.wired.co.uk/article/farming-robots-small-robot-company-tractors
Docknet v1. I trained a neural network to identify broadleaved docks in grass backgrounds/fields.
A python loop chops the original image into squares of 256 x 256 pixels, a neural network identifies if the image square contains a dock or not dock. Python then inverts the image block colours if it sees a dock. The squares are then stitched back together to form the output. The inverted blocks stand out so we can see where the weeds are. A toy example that shows the effectiveness of neural network for weed identification.
I used fastai on google colab to train and then downloaded the exported pickle file to run the model on my laptop cpu. Fastai has made something difficult so easy. Things are moving so quickly in this field.
Just ordered a Nvidia Jetson nano. Its like a raspberry pi but comes with a nvidia chip just like their graphics cards but with a lot less power, should be perfect for this application. Hopefully we will see some fast ai models running on these devices soon.
Hi Gavin,
how’s your project going? I am really excited about bringing independent tools to smaller farmers, making farming more environmentally friendly and reducing soil damage (CTF and all that) from heavy machinery.
Re. your comment about mapping drone photos to GPS- this is a solved problem that off-the-shelf products like dronedeploy will sort for you (including planning the flight).
I have been working on a (round) bale detection project (currently using YOLOv3) and getting good results.
Obviously, this is for autonomous bale collection/loading.
As an intermediate step I built a webapp that folks can upload a .gpx to. Using some clustering and TSP type algorithms it will generate a .gpx route for download and use on basically any mapping program so you can follow it “manually”.
Here’s an example:
http://fastbale.windridge.org.uk/process_file?trailer_capacity=18&filekey=532eab359e744ae7a0d4e6ff12e487e2
[Currently GAE free tier so might die if lots of people click 
]
I have family and friends playing with it and real testing come the hay season but I’d be interested in any suggestions already if farmers following the thread wants to try it! The current algorithm is not optimal and I am working on both better heuristics and also an approach where I train a DNN to learn the structure of smaller instances that I can bruteforce solve. Anyway, in simulations it already saves significant time in the field compared to a naive strategies.
Cheers, looking forward to an update!
Peter
Hi Gavin. What you and the others in this post is trying to accomplish is very commendable, and it is very exciting to follow. Using less pesticides more focused is a win/win for all parties involved, and those projects are worth backing in my experience. I have been playing around with homebuild racing drones for a bit, and have been very impressed with the power that can be pulled from very small machines. I think you might be able to combine picture collecting and small speedy drones that is tailored to your needs. I know a lot of manufactures makes great products out of the box, but if you solder your own together, with the components you need for your purpose, I think there is a lot of money and weight to be saved.
I plan on buying my parents farm within the next 3-4 years, and would like to continue farmin hay on a hobby level. When I see a person that can talk about deep learning training sets, validity, using less pesticides AND craft a buggy in his workshop I get hooked. I hope you are still working on this project as I would love to see where it leads.
Nice idea, very creative use of Yolo. What about logging your bale position / gps coordinates when you eject the bale from the round baler and then use your tool to plot the best route. Not as much fun but then you wouldn’t need the drone.
My project has slowed down a bit but I’m still thinking about it and researching most days. Finding the time to physically build stuff is my biggest hurdle. Too many project too little time. 
Hi Gavin,
Thanks, yes logging GPS at bale drop would be better in many ways. For big square bales I’ve started building a simple bluetooth switch that pings your phone when the bale drops off the back. My longer term fantasy of having autonomous loading would still need to determine orientation, so I partly did the aerial recognition as a stepping stone as I thought it would be easier from the air. And, yes fun 
However in the last week I built a simple LeNet based model (just on my laptop, no GPU) for orientation to the nearest 30 degrees from “tractor view”. It works surprisingly well and some of the “wrong” answers on the test sample are due to my bad labelling (done late at night)!
I hope to get something “working” (useful … maybe :)) for first hay cut (tractor+trailer and separate telehandler)-
I understand about lack of time! I am mainly working on the models now as it’s easy to do 10 mins labelling training material over breakfast etc.
Thats a nice ambitious target for this summer.
Im going to have to scale back my plans a little due to time. My next mini targets are to
Then I will forget about the gps control for now and just push it manually over a patch of weeds to see if I can get a trigger.
I did think about creating another dataset called “carrots on concrete” it would be a test dataset that could be used to test indoors. The pictures would be of a concrete floor using carrots or potatoes instead of weeds in a field. That way I could test all year round in relative comfort of a shed just using carrot or similar veg from the supermarket
That would be great and I’ll be interested to hear how you get on with the nano.
I think the GPS part will be easy to add on later- it could simply be something like AgOpenGPS +cheapest RTK configured to traverse the whole field.
Re. 3d printing- out of curiosity have you tried printing spray nozzles/heads?
The carrots on concrete toy setup for indoors tweaking is a good idea (I thought about doing the same with toy bales, i.e. toilet rolls :D). As a stepping stone even ping-pong balls (easy to recognise on concrete) might be worth perfecting.
Looking forward to any updates!
Re. 3d printing mount for Jetson Nano- I just saw the JetBot project ( https://github.com/NVIDIA-AI-IOT/jetbot ) - possibly the STLs could be butchered and used quickly.
Sorry for polluting your thread (maybe I should start own) but just a quick update- I expanded my training sample, switched to AlexeyAB fork of darknet, tried tweaks in activation functions. Getting much better test performance now, even on blurry stills from YouTube videos 
Changed the orientation approach so the last layer is more like a least squares regression on my guesstimated angle. Still largely based on LeNet-5 and trained on my laptop. However, I have plans to change, maybe try to modify the YOLO architecture. Expanded the sample, augmented with flips and upped size to 48x48. The model still doesn’t know 0 and 180 degrees are the same though! Test sample examples:
Built a little test rig for the Nano, Arduino and Relays. Will need to think about some weather proofing for my setup that will also allow some airflow to cool the nano.
Do you post on thefarmingforum I think I seen a similar thread there?
Looks like you are getting some good results. How accurate do you think you need to be for your automation to work.
Made this prototype to show the neural network controlling a relay. This relay is capable of controlling a spray nozzle solenoid. The camera and Jetson nano are identifying a computer keyboard below and switching the relay on when they see it. Hopefully soon I shall switch out the neural network for mine that identifies weeds. You can hear the relay switching on or look for the red LED on the relay board to see that it is working. Communication is via serial
Hi Gavin,
Im Ben, Mechanical engineer/ Sheep and Beef farmer from New Zealand, I have also been thinking about this problem from the other side of the world, though mainly looking at spraying thistles. So awesome to see your progress!! Would love to get onboard with the project in some way. do you have more details on how you trained and run your neural network? I have been building a database of thistles to do the same and would be keen to try yours out?
Cheers,
Ben
