It's not uncommon for a propeller to spin in neutral, however it shouldn't be in gear or under power (meaning, you could actually stop it with your hand... but don't try that!). Any spinning should certainly not be causing boat movement. You can double check - with the engine off and in neutral - spinning the prop by hand in either direction should produce no clicking or resistance.
If it is actually still in gear, then your shift cable has probably stretched or the linkage is fouled up. You may be able to adjust it - take a look at the linkage and double check with your service manual. It's pretty straight forward, though.