Should I connect ground to the common heatsink thingy instead?
You should use the ground shown in the data sheet. However, if your ohmmeter shows that the cut leg and
heatsink are common ("shorted" together), then the choice is yours.
I don't have the caps on yet. Still going through the pdf datasheet at fairchild.
For reliable operation you need the caps. The LM78X series tends to oscillate without them.
The caps are mentioned in the fine print of the data sheets on several of the pages. The mfgs' usually specify a minimum of .1uF on the output and .33uF on the input (low ESR caps). Attached is the exact fine print from the Fairchild data sheet.
With >7V in, the Vreg's output will be 4.9V to 5.1V. If you measure anything
else then it is defective or there is a wiring error. Although fixing up the Vreg is not going to improve your range, it will ensure reliable operation and will prevent TX2 harm. Once you get this straightened out you can work on the range problem.