As a prop/motor setup you will need something like a 100gr+ Motor with arround 800 KV and e.g. 12x4,5 CF props (for a little bit of efficiency)
Now you have 3,5 kg + Battery it will be around 5kg, so about 800gr per motor, so 1600gr max thrust per motor, not easy and efficient possible with 12" max.
My suggestion would be:
this is just illustrate what I suggest, but of course other motor/prop combinations but with similar tech. data are possible.
P.s: Did it lift off with the 11"?