Yes, I have thought of using a RC brushless motor, I have the right motor to do it too...Soda can size.
However if you think about it, it's not as practical as you might think. First of, a Spindle mean sturdy, mostly slow rpm, high toque. Our RC brushless output a lot of power, but require a lot of power as well... DC. Then again, how would it hold up to a 2 hours milling job and how long will it last at that stage... RC motor are quite and you can add water jacket for water cooled. But where am I going to find a 36-40volt Power Supply 30-40amps range cheap enough.
My brushed harbor freight router on my old woody machine has been running for 3 years.. at least 100hours of cut.
What you see in the CAD drawing is what I am going to build.. still waiting for the big bearings to come for the Spindle.