Real time synthetic view of a UAV

[clolson]

I'm probably outing myself as a computer geek here, but I wanted to mention a kind of neat thing I just got working.

I am helping out a group at the university of minnesota with a UAV project. One of the things we wanted to do as part of this project was to create a real time synthetic (computer generated) view of what the UAV *is* doing as it flies.

I'm heavily involved with FlightGear (which is an open-source flight simulator) so that was the natural choice. FlightGear has scenery for the entire world, based on SRTM terrain data, with major roads, rivers, urban areas, and most airports. It also has some limited ability to do photorealistic scenery for smaller areas (with some modeling effort.) Hopefully we can feed that back on ourselves and use imagery we capture to build 3d models of our flying area for use in our synthetic view.

Our UAV is currently flown entirely by a human pilot, but we have a MIDG-II on board (which is a IMU/INS/GPS combination.) We transmit that data to the ground with a radio modem.

I just finished some glue code that sucks in the MIDG data off the serial port, decodes it, converts it to the network structure that FlightGear expects and blasts it off to FlightGear.

FlightGear will then render/animate the scene from the UAV's perspective. The neat thing about this is that you can view the synthetic world from a cockpit perspective, a chase plane perspective, a fixed ground point perspective, and other varients can be configured.

What makes this really neat is when you start noticing that all the existing capabilities of FlightGear are still there. The cockpit instruments are *live*, altitude, speed, artificial horizon, vsi, dg, etc. Even the radios work, you can tune in virtual VOR's and the virtual needles on the virtual insruments respond as if there was actual radio equipment on board. You can turn off the instruments and turn on a HUD if you prefer. The HUD gives you a reasonable estimate of height above ground (based on the computer's terrain models which aren't half bad.) I can even imagine a day where I could lookup the terrain elevation for the UAV's position on the ground computer and send that back to the UAV so it can maintain safe ground clearance on longer flights over changing terrain.

I could even setup a virtual ILS approach to my little R/C club field (although our sensors don't have enough positional and altitude accuracy to make that a wise thing to actually try ... at least not all the way to touch down.) :-) But it might be useful for lining up an approach an maintaining a desired glide slope so you miss those pesky trees, but don't overshoot by a mile.

FlightGear puts the sun and moon in the right place in the sky for the current time so if you had live vide out of the front of your UAV running next to the live flightgear view, the sun/moon should show up in the same place on both views.

That's all cool in a geeky sort of way, but what we really want to do with this is to insert things like restricted airspace, mission goals, flight paths, important objects, etc. into the synthetic view so a ground based pilot can guide her[1] craft in a safe manner.

[1] Hey I work at a university so this is my token attempt at being politically correct so I can avoid getting shipped off to mandatory diversity training. :-) The latest thing is "ethical diversity" Ouch! Hey man, where was ethical diversity when I was president? (Sorry it's past my bed time here.)

Anyway, all this FlightGear/MIDG/radio modem stuff is working great on the ground. We are hoping to flight test on monday morning ... and if all goes well, perhaps try piloting solely from the synthetic view and wireless video imagery. We were going to try doing something similar on our last flight test, but our engine cut out midway through our initial climb out and we didn't make it back to the field completely intact. :-)

If I can come up with any interesting pictures to show after our flight test on monday, I'll see if I can post them. I need to figure out a way to put the wireless video view next to the synthetic view and package that up as a single synchronized computer movie to post ... that would be really cool ... but is probably beyond my capabilities.

Regards,

Curt.

[kd7ost]

Curt,

You do some really cool stuff. Good work on all of it. Yes, you are a geek. But thats a good thing.

Ethical diversity? What the He*l? I'm afraid to even ask but it just figures doesn't it?

Good luck on the flight testing.

Dan

[Bg~]

Curt,

I did something similar with our Piccolo based system and Open Glass Cockpit here at Penn State. Don't have much use for it yet, I just mainly did it to learn the communication protocol for the piccolo. Including Flightgear in the loop when we are flying would be cool, but the aero dept. needs to buy me some better computers before I can do it and have Flightgear running at a decent framerate :) (we just have a really slow laptop out at the field).

[djklein21]

Awesome guys, looking forward to hearing the results.

Curt, what kind of aircraft are you guys using?

[clolson]

Awesome guys, looking forward to hearing the results.
Curt, what kind of aircraft are you guys using?

Right now we are flying a Rascal 110 (v2.0) We wanted an ARF so we wouldn't need to invest a lot of time into the airframe, and we didn't know our payload requirements up front so we wanted to err on the big side. Ultimately we are going to probably end up with some combination of a PC104 stack, 3 (maybe 4) gps receivers, 2 wireless video systems, imu, flight computer, radio modem, and all the battery to power all that. We might not be able to do everything we want, but we didn't want an airframe size/power limitation. Once we nail down our instrumentation, we'll probably be able to go with a smaller airframe. On the other hand, empty space seems to have a way of getting filled up pretty quickly.

For what it's worth, if you want an airplane that just a ball to fly, take a look at the Rascal 110 ... classic golden age lines, wonderful flyer, no bad habits that I've seen, *big*, very light wing loading. We put an OS1.60FX on our first one, and oh man does (did) she perform. We have Zenoah G26 on our v2.0 and that's a bit more tame, but still more than adequate. The OS1.60FX would turn an 18x8 prop at 10,000rpm. The G26 does about 7,000rpm. So there's a big difference in top end performance, but the G26 is still plenty for scooting around the sky.

Dan has one of these (he's since moved on to a purpose built airframe) and he estimated about 60mph with the G26 and an hour flight on about 32 oz of gas. We clocked the OS1.60FX powered Rascal at 102+ mph on a downwind pass and figured top end level cruise was about 85mph. Pretty good for something that can slow down to about 15mph on landing approach (and if you have a 5-10mph headwind you get guys complaining that your landing approach is taking too long and you are backing up the pattern.) We only got about 10-12 minutes on 24 oz of glow fuel if we ran the engine hard, but a lot more if we throttled way back.

The Rascal 110 isn't perfect as a UAV platform, but it sure is a blast to fly.

Curt.

[chrisgood]

Clolson,
That tie-in from the MIDG to Flightgear sounds pretty slick. We have a MIDG in a chopper and I had thought about setting up something similar but had never gotten around to writing the code. Is your work available to poach?

BG~,
I have installed a Piccolo in a few different planes for autonomous flight. The guys at Cloud Cap are a good bunch and very helpful. The Piccolo comes with a HILSIM (hardware in the loop simulator) that ties into FlightGear right out of the box. It can be used for simulated flight before you get to the field; it is very helpful in tuning the autopilot constants before you get to actual flight. The early aero model from Cloud Cap that the user is supposed to configure for their own airplane was off a bit in the roll response loops, but they updated that in a later release. I have not tried to tie real-time telemetry or recorded flight logs into FlightGear, but considering how Cloud Cap already has the hooks in place for the HILSIM, it should not be much of a problem.

Chris

[clolson]

Chris,

Does rcgroups have some sort of PM facility? Or you can find my email address at my web site: http://www.flightgear.org/~curt/

Contact me offline and I think I can help you out (there are a couple small complicating issues, but certainly doable.)

Curt.

[clolson]

Here's an update on our "synthetic visual system" work. We fixed our radio modem interference problem so we were able to fly this morning with the live data link feeding FlightGear as our synthetic display. I had one of my buddies build a photorealistic model of our flying area and I placed it at the correct location in the FlightGear world. There is a short movie of a landing approach and touchdown on my web site, notice the live working instruments which I'm kind of proud of ... ignore the wild flying ... I'll blame that on a gusty cross wind with a lot of turbulence rolling in over the hangers. :-)

http://www.flightgear.org/~curt/Models/Special/Rascal110_2/

Movie is crappy quality, but it's enough to give you an idea of what we are doing ...

Regards,

Curt.

[LukeZ]

Curt,

I went to your site to check out the video, but the link didn't seem to work...


Luke

[clolson]

I went to your site to check out the video, but the link didn't seem to work...


Strange, the link works fine for me here (firefox/linux and firefox/windows xp.) It's just a simple .avi movie straight off my digital camera with no touch ups. Here's a direct link if that helps ...

http://www.flightgear.org/~curt/Models/Special/Rascal110_2/Movies/MVI_2986.AVI

If someone can see that I'm doing something dumb on my web site, let me know, but it should just be a standard link to a file, nothing fancy ...

Regards,

Curt.

[LukeZ]

Curt -

The link shows up now. Don't know if I had a hiccup with my browser or what... but it works fine now.

I liked the instruments!

Luke

[shedao]

I can get the file no problem. But I dont have mplayer installed so no avi support right now so ican't verify the video, but it did download.

[Bg~]

I had one of my buddies build a photorealistic model of our flying area and I placed it at the correct location in the FlightGear world.

Is there any specific software used to do this? I've never really looked into terrain building for Flightgear.

[aerogel]

very kewl idea...Huge potential in this...

[clolson]

Is there any specific software used to do this? I've never really looked into terrain building for Flightgear.

FlightGear supports a variety of 3d model formats such as openflight and 3ds. Most modeling software packages can output the models a couple different formats. And there are also model conversion utilities floating around out there. The result is that if you are careful, you can often find a reasonable conversion path that lets you use just about any modeling software you like. Creator, 3D Studio, Blender, AC3D, Medit, etc. etc.

Curt.

[clolson]

Here's a quick update on the synthetic view project. We now have live video running next to a live synthetic view on our ground station. Here are two quick screen grabs which give you a small taste. (I've never tried uploading images to this forum before so if they don't come through, you can find on them on my web page: http://www.flightgear.org/~curt/Models/Special/Rascal110_2/

Regards,

Curt.

[Bg~]

Curt,
Forgive my nosiness, but where did you pick up those detailed ground textures used in the Flightgear view? terraserver? google maps? Unfortunately for me, there is nothing high res for the State College, PA area from those sources.

It looks really sweet, BTW. Must be fun to fly. Is there much lag between the data downlink for the synthetic view and the live video? Is the latency on the displays small enough even to fly from it (disregarding downlink cutouts of course :) )?

[clolson]

Curt,
Forgive my nosiness, but where did you pick up those detailed ground textures used in the Flightgear view? terraserver? google maps? Unfortunately for me, there is nothing high res for the State College, PA area from those sources.

It looks really sweet, BTW. Must be fun to fly. Is there much lag between the data downlink for the synthetic view and the live video? Is the latency on the displays small enough even to fly from it (disregarding downlink cutouts of course :) )?

Until we get a better system in place, we borrowed imagery from the web. We are hoping to eventually be able to build environments from our own imagery, but we aren't quite there yet.

The big problem right now for us is data rate from our uav via our radio modem. I believe we are using an Aerocomm radio modem and we see evidence that packets are retransmitted several times (i.e. we get the same data a couple times) before it moves on to the next data. So instead of getting the hoped for 50hz, we are probably getting 3-5 hz on average with occasional 1/2 second pauses, lots of repeated data, or even data that goes back in time to the previous time step. Yuck! If any one has any better suggestions for radio modems that have > 2 mile range, I'd be interested.

We tried to fly from the synthetic view on a really windy turbulent day, and combined with the low data rate and dropouts, we felt it was hopeless. On a calmer day in conjunction with the video, it may be possible. Ultimately it makes sense to put some sort of autopilot onboard that can either augment stability or fly waypoints and hold altitude so the ground based pilot doesn't have to worry about wiggling the control surfaces at the right time and can pay attention to higher level stuff.

Fun stuff, but every small step forward is a major challenge!

Curt.

[Bg~]

The MHX-910 (www.microhardcorp.com) modem in the Piccolo has performed flawlessly for us. We've had it out to 1.5 miles with good signal. (I don't have to deal directly with the radio modem as its all one complete package, so your mileage may vary).

[clolson]

I hope I don't get in trouble for this, but I thought I'd post a couple movies of our stuff in action. See my post in this thread on 10/29/05 for a couple still pictures.

On 10/26/05 we did a test flight of our UAV (manually flown) and captured the live video stream from the onboard camera as well as the data stream from the onboard IMU/GPS/INS unit. We replayed the data stream in FlightGear with overlayed instruments (the synthetic view can also be drawn in real time as the data is captured) and saved that out as a movie. Then we edited the two streams together in two different ways: side by side, and blended overlay. The result is kind of interesting. You can see at the start of the flight where the IMU was pretty far out of whack ... as much as 5-20 degrees off in yaw. But as the flight progresses you can see the error diminish and later in the flight the match can be quite good (and in places it drifts off again.)

Both videos are about 50Mb to download. You will need DivX6 to play them. Linux or open source people might find they can play these with xine or mplayer. Windows people might need to go download the DivX6 runtime codec from http://www.free-codecs.com/download/DivX6.htm.

http://www.flightgear.org/~curt/Models/Special/Rascal110_2/Movies/chapt1-divx6.avi
http://www.flightgear.org/~curt/Models/Special/Rascal110_2/Movies/chapt3-divx6.avi

The synthetic view is drawn with FlightGear, an open-source flight simulator (software that is available to anyone for free.) Aircraft is a Sig Rascal 110. IMU/GPS/INS is a MIDG by Microbotics. Video is by black widow with a really lazy antenna aimer. Wireless data is done with Aerocomm radio modems.

Obviously a lot of warts ... we can/need to do a lot better with the video side of things, we need to work on reducing vibration of our camera, and we have put very little effort into video camera vs. synthetic view parameter calibration, etc. etc. but hopefully this is a little interesting to a few people. If nothing else, you can see places where the video craps out but the synthetic view tracks well. And you can see places where the synthetic view is off by quite a bit, but video is good. And it's interesting to be able to overlay live working flight instruments on the display. You could imagine that we might be able to place a depiction of restricted airspace in the synthetic view, your mind might even wander a bit futher and ponder if it might be possible to click on the screen and get a location of the point back, elevation of a mountain peak, (or maybe an address or street/highway name ...) hmmm.... :-)

Curt.

[Vindication]

Curt,

Looking very nice. Am I imaging or is your psi estimate actually pretty good? How'd you do that? I imagine there was like no wind when you were flying???

Vind

[Vindication]

Also,

Can you tell me what the pricing runs like on that MIDG?

TY

[clolson]

Looking very nice. Am I imaging or is your psi estimate actually pretty good? How'd you do that? I imagine there was like no wind when you were flying???


I assume by "psi estimate" you mean yaw/heading? That is coming straight off the MIDG. The first few seconds of the take off are clipped, but if I showed you that, you'd see that the MIDG is about 45-60 degrees off in yaw at the start. Once you fly for 30-60 seconds it seems to get within about 5 degrees. After another minute or two it is pretty close to dead on and the visual discrepancies seem like they are more due to gps errors and camera view vs. synthetic view calibration mismatch.

The midg is pretty expensive for what you get ... probably in the neighborhood of $5000-6000. As I understand it, it's based on solid state accelerometers and gyros so the sensors themselves aren't that great. The value (supposedly) is in the onboard processing and calibration.

I'm only paid 3 days a month to work on this stuff so even when I throw in my own time and dip into time from my other projects, progress is really slow ... we also have an xbow unav unit which should perform pretty comperably to the MIDG (if not better) and is more accessible. But we haven't gotten far enough with it to test fly it. The xbow unav is about $2000 if you include the stargate board (you need something to crunch the kalman filtering and that is about as good as anything in terms of size, weight, power requirements, etc.)

The on-the-ground-wiggle-tests with the unav seem promising. And it seems to start out with a better psi estimate than the MIDG ... the unav really isn't a polished and neatly wrapped package yet though ... I think it has good sensor technology underneath, but you need to do some coding to get the most out of the unit (that's good if you are comfortable with coding, and bad if you aren't a software guy.)

Oh, for what it's worth, we went out and did a big demo yesterday and our data link failed. Grrr ... I really hate surprises on demo day. I fear it will turn out to be a stupid software bug, but this was the coldest we've ever flown our data-link + synthetic view so we may have hit some temperature issues. The MIDG was still spitting out data, but all (and I mean *all*) the checksum values were wrong. This has never happened before in many, many test flights, and everything worked when we got back to the toasty warm lab. Still it worries me that data was flowing, but all the check sums were wrong (so all the packets were getting discarded). Turn of the check summing and we were getting mostly good data. I'm still confused ... smells like something dumb on my end in software (did we hit some odd condition that caused us to add up the check sums wrong?), *but* it only failed that one time in the really cold conditions ...

Curt.

[Vindication]

Curt,

That problem sounds interesting. I have noticed similiar type of behavior from AP's when they are not getting enough power. How cold was it when you were flying? Perhaps it was so cold you were getting low voltage from the batts. Last time I went to the field I actually brought a heat gun so I could warm anything up if I need too... Also if you can insulate your batteries somehow, that could help you in flight. I have actually had days where I couldn't get my rc planes off the ground because the batteries weren't giving enough juice due to the cold.

On the psi estimate- The psi reading from GPS represents groundtrack... the camera is with relation to actual heading. So on a windy day these two values can be pretty far off. I'm guessing you guys aren't doing anything special to account for this? [And your psi was so good because there wasn't wind that day..?]

Vind

[clolson]

On the psi estimate- The psi reading from GPS represents groundtrack... the camera is with relation to actual heading. So on a windy day these two values can be pretty far off. I'm guessing you guys aren't doing anything special to account for this? [And your psi was so good because there wasn't wind that day..?]


Actually we do try to get actual heading of the aircraft, not ground track. The sensor unit we are flying is called a "MIDG-II". This has 3-axis accelerometer, 3-axis gyro, 3-axis magnetometer + gps.

The unit can use gravity as a reference for roll/pitch, but there is no good reference for yaw (and like you say, ground track could be way off.) However, the unit does some sort of mathemagic to factor in the accelerometer readings over time (and maybe the magnetometer readings?) to help align the yaw estimate. We were flying on a moderately windy day and the MIDG's magic seemed to work just fine. I suspect that the gps ground track factors into the psi/yaw estimate minimally if at all.

So yes, once the MIDG zeros in on it, we should know the true heading of the aircraft pretty close -- and that is confirmed by the reasonably decent match between our video and synthetic view.

Curt.

[Vindication]

Ah.. yes I was forgetting about magnetometers. w/o them the gyros will just drift and you cannot get a good heading.

[Or that is to say you can get a good groundtrack but not a good heading].

[hugo_vincent]

The big problem right now for us is data rate from our uav via our radio modem. I believe we are using an Aerocomm radio modem and we see evidence that packets are retransmitted several times (i.e. we get the same data a couple times) before it moves on to the next data. So instead of getting the hoped for 50hz, we are probably getting 3-5 hz on average with occasional 1/2 second pauses, lots of repeated data, or even data that goes back in time to the previous time step. Yuck! If any one has any better suggestions for radio modems that have > 2 mile range, I'd be interested.

We used Aerocomm modems (http://www.albatross-uav.org) and had similar problems. We certainly won't be using Aerocomm ever again. Our first pair of modems arrived completely dead and never worked at all (I was as if the Aerocomm firmware hadn't been loaded onto the onboard micro...). We had to buy a second pair, and found that with the default antennas, at ranges less than about 10 m, one transmitter was overloading the other receiver and corrupting the data. And over a reasonable distance (100 m), many packets were dropped, many had to be resent, and some were even corrupted. Our downlink only managed a 3-5 Hz update rate in our Open Glass Cockpit-based GUI....

Did you find any solutions?

For my next revision of the hardware, I am going to include footprints for an X-Stream modem and some kind of cell phone connection (yeah, I know latency will be even worse but we want to try long distance flights).

Cheers,
Hugo Vincent.

[clolson]

We used Aerocomm modems (http://www.albatross-uav.org) and had similar problems. We certainly won't be using Aerocomm ever again. Our first pair of modems arrived completely dead and never worked at all (I was as if the Aerocomm firmware hadn't been loaded onto the onboard micro...). We had to buy a second pair, and found that with the default antennas, at ranges less than about 10 m, one transmitter was overloading the other receiver and corrupting the data. And over a reasonable distance (100 m), many packets were dropped, many had to be resent, and some were even corrupted. Our downlink only managed a 3-5 Hz update rate in our Open Glass Cockpit-based GUI....

Did you find any solutions?


Interesting ... I played around with our aerocom's for several hours in the lab on tuesday and we are seeing frequent corrupted data, frequent retransmits, frequent check sum failures. The IMU messages have a time stamp and we will very often see messages come out of order (probably due to the aerocom's retransmitting chunks of data.) Hmmm ...

The maxstreams seem to be in a similar pricerange with similar technology. It would be interesting to hear if people have had better luck with them at high data rates (i.e. 115200 baud.)

Curt.

[Fieldruts]

Would like to chat curt, when you got the time in a pm of email... great stuff and worth the sweat you put into it...

Ken

[hugo_vincent]

Interesting ... I played around with our aerocom's for several hours in the lab on tuesday and we are seeing frequent corrupted data, frequent retransmits, frequent check sum failures. The IMU messages have a time stamp and we will very often see messages come out of order (probably due to the aerocom's retransmitting chunks of data.) Hmmm ...

The maxstreams seem to be in a similar pricerange with similar technology. It would be interesting to hear if people have had better luck with them at high data rates (i.e. 115200 baud.)

Curt.

We found that they "overload" each other's receivers when they are close to each other e.g. in the lab (not sure if they are actually overloading the receiver or somehow interfering in some way), so you might want to try some tests over a slightly longer range (we ended up leaving one attached to a computer at the other end of the lab and connected to it via telnet over the computer network). We were running them at 57600 baud.

EDIT: another possibility is that you might be overwriting the buffer? I can't remember the details but I remember something where if you don't respect the RTS/CTS etc. then it might be overwriting its internal buffer. (Excuse my ramblings... my memory fails me :)

[treehog]

I might be able to contribute next year when I know eneough how to program but am only in third year of evening course in programing so still a newbee

Have you plans to put a downloadable version on the net for others to try out

Ralf

[kd7ost]

Curt,

What would it take to run a map and GPS onboard the plane? No human interface needed. What if we could program the plane to fly at 500 feet agl and keep reading the altitude from the mapping software and adjusting it's altitude to stay 500 feet above what the map says?

Dan

[clolson]

Curt,
What would it take to run a map and GPS onboard the plane? No human interface needed. What if we could program the plane to fly at 500 feet agl and keep reading the altitude from the mapping software and adjusting it's altitude to stay 500 feet above what the map says?
Dan

FlightGear is my baby so that's the first hammer I hit every problem with. With FlightGear, if you feed in a Lon/Lat, it can look up the elevation of that location. FlightGear scenery is based on SRTM data so it is reasonbly accurate (within some limitations.)

You could fetch the SRTM2 data directly and write a little application to query elevations. There's a lot of different ways you could do this.

I guess the way I would think about it is that it would be difficult to store the entire world's elevation map onboard a UAV. So you would either need to store a small subset of it onboard (knowing in advance where you will be flying) or store the scenery set on the ground station and send the location down to the ground, lookup the local elevation, then send that back up to the UAV so it can adjust it's altitude accordingly.

You could even get fancy and do some lookahead based on your current direction of travel.

Something tells me that if you are flying near significant terrain elevation changes, you may want some sort of secondary system as a sanity check ... maybe live video would work if there would be someone monitoring it ... that way if it looks like the UAV is "getting kind of low" the operator could force a climb of another hundred feet or more just to be safe.

I guess I'm kind of meandering around your question, (too much Christmas 'cheer') :-) but if you had a gps on board, you could pass that data down to your ground station via radio modem (or run some sort of flight computer onboard.) The ground station could monitor the location of the UAV, lookup elevations from some sort of height map, and then add +500 to the result and send that back to the UAV as it's target altitude. It may be more complicated than that, but I might need to plow in a few UAV's to realize what I'm missing here. :-)

For $10,000 I could probably build you something, for $20,000 I could probably get it done in '06. :-)

Curt.