VHF radio transmissions are line of sight. Therefore the higher the antenna, the farther the theoretical transmission distance.
Here is a calculator
Play around with different heights but the height has to change a lot to make a big difference. I ran a few calculations of the working distance for your situation and another vessel whose antenna was 12 feet high. I put yours in at 6 feet, 2 feet of freeboard and 4 feet of antenna. The distance is 8 miles. Changing you antenna mounting height to 12 feet makes the working distance increase to 10 miles.
If you have issues with boats you can see and not more than 8 miles away, the problem is not the antenna height. You could have a bad antenna, a bad connection, or even a bad radio.
I would check the connections first and those are OK, I suppose I would go for a new decent antenna.
One helpful clue determining if the sub par operation is both on send and receive. In other words, if the far end hears you fine but you don't hear them, then the radio is suspect also.
Find out if you have the automated radio check feature available in your area. That service enables you do make a radio check call and then hear what you said, transmitted back. It can be on several channels or not available at all. Ask around.
The usual channels are 24, 25, 26, 27, 28,