I start by chucking the joiner in my drill and using a metal file to remove material quickly.
This is plumbers emery, found in the home stores plumbing section. This will polish up the surface.
Repeat many times untill the fit is what you want. I don't like a fight so mine are pretty loose.