Yeah, I've tried that numerous times and the drift continues.
Personally I think it has to do with the strength of the battery, as different batteries seem to cause the heli to drift differently. The amount of drift also changes as the battery drains through the flight.
I've just given up trying to get the "perfect" hover and just deal with the drift, it's not that big a problem.
What I've also found that seems to help is to zero the trim or overshoot it left before unplugging the battery from the heli.