something I have been thinking about...

[socal swimmer]

I was wondering if this would work.

an autonomous helicopter would use a gyroscope attached to a computer (either on board or connected somehow to a computer on the ground) to maintain stability. The computer would control the servos on the heli, and the motors. The helicopter would have two video cameras, about a foot apart, and facing in the exact same direction, also attached to the computer. The computer would take the two images, and by comparing them (through pretty complex algorithms im assuming), determine how far away everything is. The computer would use this for obstacle avoidance, approximate speed, height, and possible even in place of the gyro, though probably not. here are the problems i have thought of so far:

** the weight of the computer/ other equipment (i want to keep the heli relatively small)
** the programming required to analyze the two images (is it even possible?)
** the speed of computer needed to analyze the two images
** connecting the servos to the computer

so this is my question: are these problems overcome-able, or is this a crazy idea?

[Tom Harper]

SS,

Interesting idea. Similar concepts are used for driving autonomous automobiles. You might Google on IVHS, neural nets, autonomous driving. Most of this stuff is a bit large for a model helicopter.

A single camera, in motion, can glean much of this data. Select two points in an image and monitor the distance between them. If they are constrained, as is an object in a traffic lane, you can get distance and convergence rate from the rate of change in size (distance between the two points).

Modern processors and cheap memory are making video analysis a viable technology. Some of the inexpensive video monitors would be ideal for experiments. Just tap into the serial video stream. You could grab sequential frames and evaluate them on a PC. Not sure you could control a Heli with it, but I suspect it has applications in UAV.

[sammons]

To get the helicopter to be autonomous (flight wise) you need gyros, accelerometers, 3 axis compass. that's just to self stabilise and stuff. to know its height you need a downward facing ultrasonic sensor and a pressure sensor for when you are above about 5 meters. GPS will be required for updating your position prediction. so at this point you can tell your UAV where to go and it will know how to stop from falling over and how to get to where its going.

cameras, this is the sucky bit. the cameras you need to do this are usually heavy and expensive. stereo vision has been done. and can be used as a form of guidance but is not overly reliable, cheap, or fast to compute. The computation will most likely require off heli processing. unless your heli is beefy enough to carry very speedy laptop or PC-104 type single board computer. the PC-104's have got a lot faster and a lot cheaper recently, you can get a Pentium M based one that runs fast and even dual core ones.

so, bottom line, it is possible. but it will take a single man many many hours and loads of money. there are universities and private companies working on this all over the world. most with large teams of people each working on separate parts. there is Autonomous UAV competitions in USA on this stuff, look it up.

:o) Phil

[socal swimmer]

ok thanks for all the help!

i will look up those things and see where i get. Chances are i won't actually build it but it will be fun to see how close i could get it to flyable. How would i connect teh heli to and off-heli computer? would i put a small comp on the heli with some type of wireless?

I think i would have fun with this because i would have to learn a lot of different things for it, which i enjoy. just the automatic stabilization would probably keep me happy for a while...

thank you!

[HELModels]

Tom mentioned Neural Networks. This link is to a company which makes software which will spit out C code of a neural network.

http://www.wardsystems.com/

Call it what you want, but what a neural network will allow you to do is make complex correlations. U of FL has some good info on video stablilization, which is based on variances and covariances, i.e., statistical analysis of video. http://www.mil.ufl.edu/~nechyba/mav/ They used a downlink and processed the video on the ground.

I've always wanted to use an FMA copilot to write a neural network using the Ward system software. It aint cheap and how to grab specific pixels of video is beyond me.

[Unterhausen]

seems like most people go with some variation of optical flow. This means that you may not actually need stereo vision. For very lightweight aircraft, video may be the only way to go.

[vector_vortex]

I am thinking about implementing this in my system although it is a long term goal. However it is a good idea. I recently received some samples of the Avago optical mouse chips. These have onboard 32x32 sensors and DSP to calculate optical flow. They are used in optical mice but have been used by universities for optical sensors in UAV applications with modified lenses. Check out Brigham Young Universities research, the title has something to do with canyons if I remember correctly. I want to try using these sensors then if that fails try implementing it with a cheap cmos cam and onboard processing. At the moment I am looking at DSP or FPGA's or a combination of both. However both are reasonably power hungry for a small UAV.
Good luck
Joel

[zitron]

I've done some image processing using webcams before, it's very CPU intensive. With two 640x480 images you'll need to compare 2x300000 pixels at 25 fps. That's doable on a laptop or something, but not a tiny micro controller. If you want to send the image back, you'll need some rediculous high speed downlink! You can compress the images or reduce the resolution, but then you lose resolution in terms of distance measurement.

Then there's the really tricky part: image recognition is tough! You need some serious AI to figure out that puff of smoke in front of you is safe to pass through, but not that tree. And there's the problem of getting the system to work under different lighting conditions, plus vibrations and high speed movement = blurred image that will mess up your algorithms, this is especially bad at low light when the camera needs longer exposure times.

Anyway, good luck to you, you'll need it!

-Z-

[HELModels]

I am thinking about implementing this in my system although it is a long term goal. However it is a good idea. I recently received some samples of the Avago optical mouse chips. These have onboard 32x32 sensors and DSP to calculate optical flow. They are used in optical mice but have been used by universities for optical sensors in UAV applications with modified lenses. Check out Brigham Young Universities research, the title has something to do with canyons if I remember correctly. I want to try using these sensors then if that fails try implementing it with a cheap cmos cam and onboard processing. At the moment I am looking at DSP or FPGA's or a combination of both. However both are reasonably power hungry for a small UAV.
Good luck
Joel

That is awesome! I just read a data sheet for one of these. What were the lens modifications? I havent found a relevant BYU link, yet.

[socal swimmer]

optical flow is determining distance based on how fast an object grows, correct?

After thinking about it some more, i think i am going to go with optical flow. It should be a lot easier. I will probably start with a car, then move to maybe a plane (there are some contests that look like fun to enter with that type of system).

when you implement a neural network, you still use one (or two) processor(s), correct? not 100s?


would a neural network be easier to use with optical flow than a normal program?

[HELModels]

I think youre confusing a neural network with a computer network, or even a parallel processing arrangement, like Beowulf. From what some say, you might actually need a Beowulf to handle the video processing. I dont think it would be that intensive. Neural networks started as an alternative to von neumann architectures, but are now mimiced by software. They wanted to do electronically what biological systems do naturally, learn. NN's have layers which when presented with a pattern of data, a response is triggered in the final layer by one or more neurodes. The correct response is triggered even when presented with unfamiliar patterns.

Those optical mice already can detect direction and speed of motion which is why they call them navigational devices. The big trick seems to be getting a useful image into them. They use an led to illuminate a surface that is only about 2 mm from the principle distance of the lens. An image from another source would need to be bright enough, the right size, and in the right part of the spectrum. They can process 1500 fps and 500 cpi, while 30 fps looks normal to human vision.

[Unterhausen]

NN's have layers which when presented with a pattern of data, a response is triggered in the final layer by one or more neurodes. The correct response is triggered even when presented with unfamiliar patterns.
You must be a lot better at training neural nets than I am.

[hg1]

Check out the Analog Devices Blackfin BF561 processor - it's dual core with dual digital camera interfaces. Each core processor runs at 600MHz, and the ALU is actually dual path, so it does 2x integer operations per cycle, so the entire processor is capable of 2400 integer MIPS at 10% the power consumption of a Pentium. http://www.bluetechnix.at builds some BF561 eval boards with dual camera interfaces.

[HELModels]

You must be a lot better at training neural nets than I am.

I know what I "expect" from one and I know what I would feed it. I would leave the self organizing and training part to the network. You, sir, are "probably" far more qualified, experienced and better funded than I ever "expect" to be.

[vector_vortex]

The BYU paper can be found here
http://www.ee.byu.edu/faculty/beard/papers/preprints/BarberGriffithsMcLainBeard05.pdf
http://www.ee.byu.edu/faculty/beard/papers/thesis/DaveJohanson.pdf
This one mentions the Agilent Sensor but they have since been spun off into Avago.
Another interesting stereo + optical flow system
http://cres.usc.edu/pubdb_html/files_upload/414.pdf
have fun
Joel

[socal swimmer]

from what i have seen here, optical flow is usually used on teh sides. will it work when forward-facing?

assuming i used low-resolution video camera(s) (640*480), how much computational power would be needed (would a 1.25GHz PowerPC G4 be fast enough)?

I think that if i do that i will start on a car, and then move to a heli (because of a supreme lack of safe testing grounds for a heli).

thanks for the help. its been great!

[vector_vortex]

I think the reason that people use the side facing camera is that it means you don't have to worry about the scaling effects of optical flow. Imagine if you are looking out the side of your aircraft objects should not appear to get bigger or smaller, they stay the same size and move across your field of vision. Now if you are looking out the front you have problems trying to match objects in the previous frame to the current one (pretending you're a computer) because the objects will appear larger as you approach them. I think the approach here is to use stereo vision to collect the 3D spacial information and then optical flow once the 3D has been combined. Just my thoughts though.

In terms of CPU power though I don't know. I would envisage that you could do it on that machine. but you would need to consider overheads from the OS and things if it is a pre existing computer. The altivec engine in the G4 would accelerate things but I feel that in this situation a dedicated Digital Signal Processor software approach or hardware FPGA approach would be best. That said there is nothing wrong with prototyping with matlab or similar.
Have fun and let us know how it goes
Joel

[HELModels]

I just found some software called roborelm that has an optical flow module. I dont have a clue how to get the vector from it, but I used it on a video of the UFL's horizon algorithm. The vector followed the horizon orientation and the length grew with rate of turn.

[zozie]

It is certainly possible to use only one camera for obstacle recognition.
This uses only one camera...

http://www.youtube.com/watch?v=DJNDKFg8LDA&feature=channel_page
And here, one camera with an Ultrasonic range sensor to follow and object and keep a set distance.

http://www.youtube.com/watch?v=WXnhJUIcBrU&feature=channel_page
I used the Aforge library with C#

Keep in mind tho this only uses an Intel Atom. When I simulate this stuff on my desktop everything MUCH smoother.