The screw at the end is to hold a spinner on.
Imagine the collet and inside of the nut as having matching tapers. When the "nut" (the proper name for the nut would "collet closer") is forced back towards the motor and larger end of the taper, it causes the fingers of the collet to clamp down on the shaft.
If the nut is not going back far enough to close the fingers, it is probably because you also need to have a washer-like piece (called the prop hub/collet closer in most cases) between the back of the prop and the nut or closer.
Then, there would be another washer on the front of the prop, then the nut goes on the exposed thread on the end of the prop adapter. You can see all the parts on the adapter's shaft/collet here.
The parts are, from left to right, prop nut > prop washer > prop hub/collet closer > collet gripping fingers
When the nut on the front of the adapter is tightened it forces the washer, prop hub, and prop hub/collet closer back twoards the motor and the tapers cause the fingers to grip the shaft.
You want to take all the slack out with you fingers, hold the prop blades and prop hub/closer against turning, and then tighten the nut to take all the play out. When you use a wrench, you only want to apply a light clamping force or it will stretch the aluminum where the threads are and break the shaft.
So tighten it lik 1/8th of a turn and see if the collet slips on the shaft. it if does, only tighten 1/8th turn more and test again. When it will not slip you are done.
If the shaft is highly polished you can take 300-400 grit emery cloth and slide it lengthwise down the shaft to create some small striations on the shaft. That will improve the collets grip on the shaft.
Also, degrease the shaft and inside of the collet with alcohol or acetone or a good degreaser.