First, you need to understand that the autopilot will try to keep circling above home within the radius set on RTH loiter radius
, note that can be set to clock wise(CW) or counter clock wise(CCW). So if you are flying inside this radius the autopilot will effectively try to go out of home. Also even as it is turning slowly, if you leave it as is, it will eventually go back home by doing a very wide turn.
If you still want to increase the autopilot authority, Increase Max roll angle
, make sure that if you move the ailerons max to the right the autopilot set this max angle on FBW mode. If not increase the roll proportional gain
on slow increments until roll axis start to oscillate and then set it 5-10 units back.
If it does still turns slowly, increase heading proportional gain
on slow increments until it does not improve anymore, if you reach this adjust and still need more bank angle then increase heading proportional limit
Finally... I have explained this few times on the rangevideo forum. If you feel frustrated anytime a quick use of the search function there, will take you out of frustration.